ParseBench-2 introduces severity-aware metrics for document parsing
LlamaIndex released ParseBench-2, a benchmark with 4,317 test cases that evaluates document parsing across six dimensions using severity-capped scoring.
LlamaIndex has released ParseBench-2, a comprehensive benchmark designed to evaluate the accuracy of document parsing systems. Published on November 12, 2026, this update expands on its predecessor by introducing severity-aware metrics and new datasets targeting complex real-world documents. The release provides developers building retrieval-augmented generation (RAG) pipelines with a more rigorous standard for measuring how well their systems handle text, tables, charts, and forms.
What happened
ParseBench-2 combines broad coverage of real-world documents with a new scoring methodology that prioritizes critical errors over minor formatting issues. The benchmark consists of 4,317 page-level test cases derived from 2,645 source documents. These cases include revised pages from ParseBench-1 along with 2,293 new examples specifically chosen to challenge parsers with difficult layouts, dense tables, and government forms. The dataset is hosted on Hugging Face and includes an open-source evaluation suite called parse-bench.
The update introduces six capability dimensions: text, formatting, tables, charts, visual grounding (layout), and forms. A significant addition is the Forms dimension, which covers 600 form pages, including real government documents with typed fills and blank fields. The benchmark also adds targeted subsets for complex table headers, financial tables, insurance filing tables, and dense tables. This expansion addresses common failure points in production systems where structural relationships and specific data values are crucial for downstream tasks.
How it works
The core innovation in ParseBench-2 is its use of severity-aware metrics. Instead of simply calculating percentage accuracy, each page score is determined by the lower of a parsing score and a severity cap. This means that even if a parser gets 99% of a page correct, a single critical error—such as a flipped negation or a misread digit in a financial table—will significantly reduce the score. The metrics are calibrated against 330 controlled scenarios across six severity test suites, categorizing errors as critical, major, minor, or benign.
The evaluation code suite ships with the parse-bench package, allowing developers to run pipelines on every subset and generate detailed reports. The system checks for over 140 error types, including changed facts, broken structural relationships, missing content, and invented content. For example, in the tables dimension, a swapped header is considered a critical error, while extra HTML markup is treated as a benign variation that does not penalize the score. This approach ensures that the benchmark reflects the actual impact of errors on business logic and user trust.
Key details
- Dataset size: 4,317 page-level test cases from 2,645 source documents, including 2,293 new hard cases.
- Six dimensions: Text, formatting, tables, charts, layout (visual grounding), and forms.
- Severity-aware scoring: Metrics are capped by severity levels, ensuring critical errors like misread digits or flipped negations heavily impact scores.
- New form tests: Includes 500 real government form pages and 100 fillable-form pages, checking for accurate field values and empty field handling.
- Open-source tooling: The
parse-benchpackage allows users to install via pip and run evaluations locally, generating per-page CSV and HTML reports. - Non-comparable scores: Scores from ParseBench-2 are not comparable to ParseBench-1 due to revised annotations and new metric definitions.
Why it matters
For engineers building RAG systems, document parsing is often the weakest link in the chain. Traditional benchmarks may overlook subtle but catastrophic errors, such as misinterpreting a "not" in a legal clause or swapping rows in a financial table. ParseBench-2’s severity-aware metrics provide a more realistic assessment of these risks. By highlighting critical failures that benign accuracy metrics might miss, it helps teams prioritize improvements that actually matter for product reliability and user safety.
The inclusion of specialized datasets for financial and insurance tables, as well as government forms, addresses specific pain points in enterprise applications. These documents often have complex structures, merged cells, and hierarchical headers that confuse standard parsers. By testing against these specific challenges, developers can identify weaknesses in their OCR and layout analysis components before they reach production. This leads to more robust systems that can handle the messy reality of scanned documents and multi-column layouts.
What you can do
- Install the evaluation suite using
pip install "parse-bench[runners,fast]"to start testing your current parsing pipeline. - Download the dataset to
./data_parsebench-2and run the default scorers to establish a baseline performance score. - Review the per-page HTML reports to identify specific error types, such as missed headings or incorrect table cell associations.
- Compare your results against the leaderboard by submitting your scores via a pull request to the model’s Hugging Face repository.
- Use the targeted subsets, such as
financial_tablesorforms, to focus optimization efforts on the most critical document types for your application. - Extend the evaluation to your own custom parsers by formatting outputs according to the ParseBench results layout and running the evaluation command.



