Research question
When a PDF becomes HTML, how much of its meaning survives: heading hierarchy, table header relationships, reading order, link destinations, and image alternatives?
Overview
Document AI benchmarks score markdown text similarity. PDF checkers score tag conformance. Web checkers score generic page rules. None of them measure whether a converted document is semantically faithful and accessible, which is the only question that matters to a screen reader user reading a converted notice, statement, or report.
This benchmark scores conversions on outcome dimensions: heading tree accuracy, table cell and header-association accuracy, reading order, link text and destination survival, list boundaries, and image inventory with purpose decisions. It runs on a stratified public corpus of 50 government documents across ten difficulty classes, from simple linear notices to scanned archival records.
The corpus, per-document evidence, and scoring method are already public on this site. The benchmark phase adds labeled ground truth and scores independent conversion tools alongside our own pipeline, and every claim ships with its measurement method.
Why this matters
The measurement gap is the market gap
In 2026, document parsing leaderboards are saturated with markdown fidelity scores while regulators state plainly that automated remediation is not yet reliably trustworthy at scale. The missing instrument is a benchmark that measures accessibility outcomes of conversion. Whoever measures honestly, in public, sets the bar for everyone.
Buyers cannot compare tools today
A government web team choosing between remediation vendors has no independent yardstick. Vendor accuracy claims are self-reported and usually unfalsifiable. A public benchmark with a frozen method and published per-document evidence gives buyers something checkable.
What we measure
- Heading fidelity
- Level accuracy per heading and exact tree match per document
- Table semantics
- Cell boundary accuracy plus header-association accuracy for row and column headers, including spans
- Reading order
- Pairwise order accuracy and normalized edit distance against labeled order
- Links
- Anchor text and destination survival for printed URLs, link annotations, and internal references
- Images
- Inventory recall and decorative-versus-meaningful purpose accuracy
- Routing safety
- False-safe rate: documents routed to automation that needed a specialist
Method
Corpus: the 50-document public conversion pilot, ten classes, five documents each, acquired only from official publisher URLs with recorded hashes and malware scans. Source PDFs are never redistributed.
Ground truth: two independent human labelers per document for headings, order, tables, links, and image purpose, with disagreements adjudicated and recorded.
Scoring: the frozen automated source-fidelity method already published with this corpus, extended with the labeled semantic dimensions above.
Subjects: our automated pipeline plus independent open-source and commercial converters, each run with documented settings.
Publication: per-document scores, method version, and raw evidence, updated when the method or corpus version changes.
Evidence so far
Only recorded findings appear here. Anything not listed has not been measured yet.
Corpus and automated evidence are live
The 50-document corpus explorer publishes source-fidelity, visual-coverage, and markup evidence for every document today.
The method already catches real defects
An August 2026 A/B run of two pipeline versions over this corpus caught a text-loss defect caused by malformed ActualText tagging in a federal fact sheet before it reached production.
Questions we expect
How is this different from OmniDocBench or olmOCR-bench?
Those benchmarks score text and layout similarity of parser output, mostly as markdown. They do not score heading levels, table header associations, link destinations, or alternative text, because their ground truth does not contain those dimensions. This benchmark exists specifically for them.
Will your own pipeline be scored by the same rules?
Yes, and its failures will be published with the same detail as its wins. The August 2026 A/B report on this site already documents both.