Structured Document Extraction
This benchmark evaluates document extraction platforms that take a user-defined schema and return structured records.
Benchmark overview
What is included and excluded
Evaluates whether a document extraction platform can turn a user-defined schema into correct structured records across layouts, scans and repeated sets, while staying honest about what it could not find.
A buyer learns how far the platform can go in turning documents into usable structured output, and whether it clearly separates extracted facts from missing ones or values found elsewhere in the document.
In scope
- User-defined schema to structured records from many documents
- Scalar fields and repeated or nested records
- Digital, scanned and mixed documents
- Training from labelled examples
- Correcting returned data
- Working with extracted records: filtering, totalling and asking questions
Out of scope
- Document classification
- Packet splitting
- Routing document types to different schemas
- Transcription quality
- Non-English documents
Participating tools
How each tool did on the 19 scenarios in this benchmark. Each cell is one scenario, in the same order for every tool and grouped by capability; Pass and Fail come from published Results.
| Tool | Scenario results · 19 scenarios in 7 capabilities | Passed | Failed | Explore |
|---|---|---|---|---|
| FutureSmart Document Intelligence | 15of 19 graded | 4 | View tool → | |
| Landing AI | 11of 14 graded | 3 | View tool → | |
| Nanonets | 11of 12 graded | 1 | View tool → | |
| LlamaParse | Failed: Corrected data goes downstream | 10of 11 graded | 1 | View tool → |
| Extend AI | Failed: Aggregate across documents | 9of 10 graded | 1 | View tool → |
| Docsumo | 6of 6 graded | 0 | View tool → | |
| Reducto | 1of 1 graded | 0 | View tool → | |
| 3 more toolsin the roster | No published Results yet for Datalab · Retab · Unstract.Each tool gets its own row here once its first Result is published. | |||
Capabilities & scenarios
19 scenarios grouped by 7 capabilities. Open a group to explore its scenarios in this benchmark.
Field Extraction5 scenarios
Returns the scalar fields the schema asks for — present, interpreted, derived, absent or differently laid out — correctly typed
Capability in this benchmark → · Global definition →
- The value is directly available4 tools with published Results
- The value needs a supplied definition3 tools with published Results
- The value must be derived4 tools with published Results
- The same field across layouts4 tools with published Results
- The field is absent5 tools with published Results
Nested & Repeated Fields4 scenarios
Returns repeated sets of records completely, with each value under the right record: none dropped, none invented, none cross-bound
Capability in this benchmark → · Global definition →
- A repeated set of records4 tools with published Results
- The set continues across a page break4 tools with published Results
- The set contains a non-record4 tools with published Results
- The set is legitimately empty6 tools with published Results
Training1 scenario
Learns from labeled examples — documents paired with their expected output — and applies what it learned to a document it hasn't seen
Capability in this benchmark → · Global definition →
- A field the tool can only get right from examples1 tool with a published Result
OCR2 scenarios
The same schema returns the same values when the document is an image, and a mixed document is handled page by page
Capability in this benchmark → · Global definition →
- A cleanly scanned document4 tools with published Results
- A document mixing digital and scanned pages5 tools with published Results
Source Grounding2 scenarios
Points to where a returned value came from — and attaches no pointer to a value that wasn't there
Capability in this benchmark → · Global definition →
- Trace a value to its location4 tools with published Results
- The field is absent5 tools with published Results
Review and Correction3 scenarios
A human fix reaches the stored record and the systems downstream of it
Capability in this benchmark → · Global definition →
- Correct a field value1 tool with a published Result
- Repair a record3 tools with published Results
- Corrected data goes downstream4 tools with published Results
Querying3 scenarios
Operates over the extracted records — filters, totals, answers questions — and is clear when an answer didn't come from them
Capability in this benchmark → · Global definition →
- Filter records across documents3 tools with published Results
- Aggregate across documents3 tools with published Results
- The answer was never in the schema3 tools with published Results
Results overview
Current published evidence in this benchmark.
Publication availability is separate from test coverage and unpublished research progress.
How the benchmark works
A public summary of the evaluation method. The same defined scope and evidence standard apply to every tool assessed under this version.
Unsupported capabilities
A capability a tool doesn't support scores 0, and the 0 stays in the denominator. No tool ranks higher for having less product.
Resources and fixtures
The registered material and systems that create a consistent test environment for this benchmark.
Invoice, receipt and bank statement test documents
A fixed round-1 fixture corpus of 19 fictional PDFs for Structured Document Extraction (R5 v1).