Tool in benchmark · Version 1

Mistral AI in Converting a complex PDF into clean Markdown with a hosted API

Scenario-level performance from current published Results.

7Passed
2Failed
1Different test set
2No published Result yet

10 of 12 scenarios published

CapabilityScenariosResult
Code Extraction0 pass · 1 fail · 1 of 1 published
Equations & Mathematical Notation1 pass · 0 fail · 1 of 1 published
Figures & Charts1 pass · 0 fail · 1 of 2 published
Heading & Section Structure1 pass · 0 fail · 1 of 2 published
Reading Order & Layout1 pass · 0 fail · 1 of 1 published
Scanned Document OCR2 pass · 0 fail · 2 of 2 published
Table Extraction1 pass · 0 fail · 2 of 2 published
Text Fidelity0 pass · 1 fail · 1 of 1 published
PassedFailedPublished, different test setNo published Result yet

Where Mistral AI failed

2 of 9 graded scenarios. Open a Result to see the conversation and the evidence.

Code Extraction

A document containing a code block

Mistral AI returned code without fences and flattened its nested indentation when converting the PDF to Markdown. The Python code was no longer runnable, and the shell command lost its continuation indentation.

View Result →
Text Fidelity

An ordinary digital text document

Mistral AI did not keep ordinary digital text complete and unchanged. The returned Markdown preserves the body prose, but page furniture lands inside the text at four page breaks, and the raw page markdown shows the same split.

View Result →

How Mistral AI performed

Open a capability to explore its scenarios. Each row reports the test set in its published Result.

Code Extraction1 fail
ScenarioPublished outcomesTest coverageResult
A document containing a code block
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

Mistral AI returned code without fences and flattened its nested indentation when converting the PDF to Markdown. The Python code was no longer runnable, and the shell command lost its continuation indentation.

1/1 assessed1/1 gradablePublished test setView Result →
Equations & Mathematical Notation1 pass
ScenarioPublished outcomesTest coverageResult
A document containing mathematical equations
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Mistral AI kept the mathematical equations as LaTeX math in the Markdown for this PDF. The returned Markdown preserved all six display equations as math blocks with tags, kept inline symbols inside math delimiters, and carried through the page-break text and running heads. The page-by-page comparison sheets matched the source pages shown there.

1/1 assessed1/1 gradablePublished test setView Result →
Figures & Charts1 pass1 no Result yet
ScenarioPublished outcomesTest coverageResult
A document with figures and captions
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Mistral AI preserved the figure at its source-page position and kept the caption directly below it. The Markdown places `![img-0.jpeg](assets/img-0.jpeg)` between the two surrounding paragraphs, with the Figure 1 caption immediately beneath it.

1/1 assessed1/1 gradablePublished test setView Result →
A document with a data chartNo published result
Heading & Section Structure1 pass1 no Result yet
ScenarioPublished outcomesTest coverageResult
Footnotes at the bottom of the page
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Mistral AI keeps footnotes out of the body flow and links each marker to its note. The Markdown output places 13 footnotes after the page bodies in source order, and the 26 marker occurrences each appear once inline and once at their notes, so the association is preserved.

1/1 assessed1/1 gradablePublished test setView Result →
A document with styled headings and subheadingsNo published result
Reading Order & Layout1 pass
ScenarioPublished outcomesTest coverageResult
A page laid out in multiple columns
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Mistral AI read the two-column page down each column in turn rather than across the page. On the three pages, the left and right columns stayed in order without interleaving.

1/1 assessed1/1 gradablePublished test setView Result →
Scanned Document OCR2 pass
ScenarioPublished outcomesTest coverageResult
A cleanly scanned document
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Mistral AI accurately recovered text from a clean scanned document. It placed the results table’s first data row, the sampling point, in the Markdown header without losing text.

1/1 assessed1/1 gradablePublished test setView Result →
A document mixing digital and scanned pages
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Mistral AI processed the scanned page in a mixed digital and scanned document rather than skipping it. The page appeared in document order, with tables in the same style as the digital pages.

1/1 assessed1/1 gradablePublished test setView Result →
Table Extraction1 pass1 different test set
ScenarioPublished outcomesTest coverageResult
A simple, clearly formatted table
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Mistral AI preserved a simple, clearly formatted table: the returned Markdown kept the four headers in order, retained eight data rows, and matched all 32 data cells against the source. The table stayed a single pipe table, and the comma inside "Metals suite, eleven elements" did not split the cell.

1/1 assessed1/1 gradablePublished test setView Result →
A table that continues across a page breakDifferent test setPublished Result available, but not included in this comparison.View Result →
Text Fidelity1 fail
ScenarioPublished outcomesTest coverageResult
An ordinary digital text document
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

Mistral AI did not keep ordinary digital text complete and unchanged. The returned Markdown preserves the body prose, but page furniture lands inside the text at four page breaks, and the raw page markdown shows the same split.

1/1 assessed1/1 gradablePublished test setView Result →

Reading these Results

Published evidence and test coverage answer different questions.

Publication availability

Which scenarios have a Result?

A published Result is public evidence for this tool on one scenario. “No published result” does not say whether testing has taken place.

Test coverage

What does each Result cover?

Assessed includes Pass, Fail and Not gradable. Gradable includes Pass and Fail. Both use the pinned test count in that published Result.

Scenario scope

Out of 12 scenarios

The summary counts each of the 12 scenarios once. A scenario fails when any of its published test cases failed. “No published Result yet” means nothing is public for that scenario; it does not say whether Mistral AI was tested on it.

Mistral AI in Converting a complex PDF into clean Markdown with a hosted API | AI Demos