Scenario in benchmark · Version 1
A document containing a code block
A document contains a code block.
How the tools performed
Every in-scope tool is visible. Outcomes come from current published Results for this scenario and benchmark version.
Publication availability: 6 tools have a current published Result.
No published result does not tell you whether a tool has been tested. Test coverage is shown only for comparable published Results.
| Tool | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Published Results · alphabetical, not ranked | |||
| Adobe PDF Extract API | 0 Pass1 Fail0 Not gradable 1 of 1 test case failed | 1 of 1 assessed1 of 1 gradable | View Result → |
| Extend AI | 0 Pass1 Fail0 Not gradable 1 of 1 test case failed | 1 of 1 assessed1 of 1 gradable | View Result → |
| Firecrawl | 0 Pass1 Fail0 Not gradable 1 of 1 test case failed | 1 of 1 assessed1 of 1 gradable | View Result → |
| Mistral AI | 0 Pass1 Fail0 Not gradable 1 of 1 test case failed | 1 of 1 assessed1 of 1 gradable | View Result → |
| Nutrient.io | 0 Pass1 Fail0 Not gradable 1 of 1 test case failed | 1 of 1 assessed1 of 1 gradable | View Result → |
| Reducto | 0 Pass1 Fail0 Not gradable 1 of 1 test case failed | 1 of 1 assessed1 of 1 gradable | View Result → |
| No published result · alphabetical | |||
| GPT-5.6 terra | — | — | No published result |
| Landing AI | — | — | No published result |
| LlamaParse | — | — | No published result |
| PDF Vector | — | — | No published result |
| Tensorlake | — | — | No published result |
| Unstructured | — | — | No published result |
| Upstage AI | — | — | No published result |
Assessed = Pass + Fail + Not gradable. Gradable = Pass + Fail. Both use the published Result’s pinned-test denominator. — means not publicly available.
Test design
- Pinned test cases
- 1
- Disclosure
- 1 public · 0 withheld
- Capabilities exercised here
- Code Extraction
What this scenario evaluates
- Whether the code remains a distinct preformatted or fenced block.
- Whether line breaks and indentation are preserved.
- Whether the code is not reflowed into ordinary prose.
- Whether the code block is still recognisable as code rather than flattened text.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the results are graded
- Pass: the test-case expectations hold.
- Fail: an expectation demonstrably does not hold.
- Not gradable: the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned.
Benchmark methodology →