Research question
When two reputable PDF/UA validators examine the same document, where exactly do they disagree, and what does that mean for anyone whose compliance evidence is a checker screenshot?
Overview
Independent research on 155 real files found leading PDF/UA validators disagreeing in half of cases. That headline number deserves a checkpoint-level answer: which Matterhorn Protocol checkpoints drive the divergence, and are the disagreements interpretation differences, bugs, or genuinely ambiguous specification text?
This study runs multiple validators across our public corpus classes, aligns findings per Matterhorn checkpoint, and classifies each divergence. The output is a disagreement map that tells practitioners which green checkmarks travel between tools and which do not.
The practical stakes: passes checker X is used as compliance evidence every day, in procurement and in legal contexts. A published disagreement map makes that evidence exactly as strong as it deserves to be, checkpoint by checkpoint.
Why this matters
Compliance evidence should be portable
If a document passes one validator and fails another, the pass communicates almost nothing. Checkpoint-level data shows which checks are stable ground and which are tool-specific interpretation.
Specification feedback needs specifics
Standards bodies can only tighten ambiguous language when someone documents precisely where implementations diverge. A disagreement map is actionable input, not criticism.
What we measure
- Per-checkpoint agreement
- Agreement rates for each machine-checkable Matterhorn checkpoint across validators
- Divergence classes
- Interpretation difference, implementation bug, or specification ambiguity, with examples
- Document-level impact
- How checkpoint divergence rolls up into contradictory pass and fail verdicts
Method
Validators: veraPDF and PAC at pinned versions, extended to others as access allows, with every version recorded.
Corpus: the public pilot corpus classes, which span tagged, untagged, scanned, and form documents.
Alignment: findings mapped to Matterhorn checkpoints; a manually adjudicated sample classifies each divergence.
Evidence so far
Only recorded findings appear here. Anything not listed has not been measured yet.
Infrastructure exists
veraPDF runs pinned in our production stack today; its verdicts are already recorded as evidence for source documents in the public corpus explorer.
The prior result is real and cited
The 50.3 percent disagreement finding on 155 files is published, peer-visible, and regularly cited. A checkpoint-level extension is the natural next study, and nobody has published one.
Questions we expect
Is the point that validators are bad?
No. Validators are essential and this study depends on them. The point is that a pass is evidence about one tool's interpretation, and practitioners deserve to know where interpretations align.