What is measured, which reader does the reading, and on what machine. The figures are not here: each one is registered with the run that produced it, and this page is the method those runs follow.
Documents the validator judges to satisfy PDF/A-2b, over the documents it judged.
This is the axis every vendor publishes, and on its own it is the easiest one to win. A validator checks the form of a document, not whether its content survived.
What is not judged: A validator that stops partway through has not returned a verdict. Those documents are counted and reported on their own line, never folded into the pass side, and a conversion the tool refused counts as a document.
How much of the source document's text is still readable back out of the converted copy, as a median and as a fifth percentile, plus how many documents fall below the threshold. Words and characters are reported separately.
The axis nobody publishes, and the one where the trade shows. Content that is gone cannot be invalid, so throwing text away raises the conformance rate. Two numbers rather than one because a broken word separator reads as near-total word loss while every letter is still there; low on words and high on characters is a separator failure, low on both is real loss.
What is not judged: A document with no source text to compare against is reported as such rather than scored, and mutool exiting non-zero on a warning does not drop the document from the set.
Pages rasterised from the source and from the converted copy and compared pixel for pixel, reported as the documents that changed visibly and as the difference at the ninety-fifth percentile.
Text can survive in the content stream while the page it draws no longer resembles the original, and neither of the first two axes sees that. The screen for visibly changed is a mean absolute difference above two percent of the grey range, far above antialiasing and far below a page that lost its text. It selects the set worth looking at with your own eyes; it is not a verdict. The page count and the resolution are printed with every run, because a rendering number that travels without them says less than it appears to.
What is not judged: A page that comes out at a different size counts as a page that changed, not as a page that could not be compared. Skipping it would drop exactly the documents where the conversion went wrong.
Bytes written over bytes read, as a total across the set and as a median per document.
The axis where we currently do worst, and it is here for that reason. An archive keeps the file forever and pays for every megabyte the conversion adds.
What is not judged: A document the converter refused contributes no output bytes and is counted as a document, so a converter cannot improve this axis by declining the hard cases.
A validator that falls over has not said the document is fine. Writing those off as unjudged is the cheapest route to a perfect score, and it is the one mistake this project has already made: a conformance figure went out in August 2026 that was high because the build underneath it was damaging content streams, and a validator cannot fully inspect content it cannot parse. The figure was withdrawn.
Conversions that failed, documents the validator could not judge, batches that died, documents with no source text to compare against — each is counted and each gets its own line. Measuring only what a tool managed measures the documents it liked.
Each axis on its own flatters a different failure. A converter that discards text wins on conformance and on size; one that gives up early wins on size and on rendering. Publishing one axis leaves the reader guessing at the other three, so all four go out or none does.
The set this conversion was tuned on gives an upper bound on our own side, and it is labelled as one. A figure meant to describe how the converter behaves on documents it has never been diagnosed against comes from the holdout. Another tool run through the same harness has seen neither, so its side of a comparison is independent whichever set it runs on.
A figure that goes outside this repository carries an identifier in the claim register, and that identifier carries the source, the run and the date it was measured. Inside the repository the published table is regenerated from the run files on every push and compared against what is written down, so a figure typed in by hand, a figure that outlived its run, and a run file deleted from under a published figure all come out as a difference.
Without veraPDF or mutool on the path, the axes that need them record that they were skipped and why, and no figure may be published from them. A gap is legible; an axis reporting a number it did not measure is not.
benchmarks/pdfa/govdocs_sample_300.txtThe set the PDF/A conversion was tuned on. govdocs is public, so the same corpus can be assembled from the names in the list.
benchmarks/pdfa/govdocs_holdout_1000.txtDocuments this project has never diagnosed a failure on. This is the set a claim about real-world behaviour is meant to come from.
corpus/GATE_CORPUS_MANIFEST.jsonDrawn from the pdf.js test suite, the Apache PDFBox test resources and the veraPDF corpus, under Apache-2.0 and CC-BY-4.0, pinned per file by SHA-256 and fetched by name rather than committed. Cheap enough to run often, and not representative of an archive: a test suite is mostly the files that once broke something — no text layer, damaged on purpose, meant to be refused. Its refusals and its retention tail read as a sample of the world only if you ignore what it is.
The lists travel, the documents do not. Names and hashes are enough to assemble the same corpus, and shipping the PDFs would republish somebody else’s files.
dev-macbook-m1proMacBook Pro, Apple M1 Pro, ten cores, macOS 26.5benchmarks/BASELINE_HARDWARE.tomlA run records the class it ran on, and the harness refuses to write a result whose class declares a core count the machine underneath does not have. The class, not the host: a pinned hostname goes stale the same way a hardcoded label does, and the label this project used before was a string literal that no run could ever contradict.
PDF/A requires embedded fonts, so a document that only referenced Helvetica gets a substitute from whatever is installed. A conformance figure measured on macOS is a floor, not the figure Linux gives. Absolute timings from a laptop are not comparable across months at all; it throttles, it sleeps, and it shares its cores with whatever else is open.
Get the readers. Both axes that need a reader use one that is not ours.
verapdf --version && mutool -vAssemble the corpus. govdocs is public; the list names the files.
cat benchmarks/pdfa/govdocs_sample_300.txtRun any converter over it. The command is yours; {in} and {out} are substituted per document.
node benchmarks/pdfa/reproduce/compare.mjs --corpus-dir <corpus> --converter 'your-tool {in} {out}' --name 'Your Tool 1.2'Or run the recurring programme, which records the machine class with the result and refuses to write one that does not match the machine.
BENCH_MACHINE_CLASS=dev-macbook-m1pro python3 scripts/benchmarks/v7_benchmark.py measure --set gate-corpus --corpus-dir <corpus>The documented PDF/A recipe, run through the same harness with the same readers, with nothing tuned for it. A different set of flags could produce a different result, and nobody outside this project has checked ours.
gs -dPDFA=2 -dBATCH -dNOPAUSE -dQUIET -sColorConversionStrategy=UseDeviceIndependentColor -sDEVICE=pdfwrite -dPDFACompatibilityPolicy=1 -sOutputFile={out} {in}benchmarks/pdfa/reproduce/compare.mjsRuns any converter over a corpus and reports all four axes. Takes the converter as a command line, so it measures ours and yours the same way.
Developed in the private engine repository and not yet in the public tree.
benchmarks/pdfa/reproduce/COMPARING.mdHow to point the harness at your own converter, what each axis does and does not cover, and the two things that make your numbers differ from ours.
Developed in the private engine repository and not yet in the public tree.
scripts/benchmarks/v7_benchmark.pyThe recurring programme: measures a document set, writes one run file per run, and regenerates the published table from those files and nothing else.
Developed in the private engine repository and not yet in the public tree.
benchmarks/BASELINE_HARDWARE.tomlThe machine classes a measurement may name, what each one is, and whether its calibration has expired.
Developed in the private engine repository and not yet in the public tree.
scripts/ci/a_published_number_names_its_run.pyRefuses a published figure that no run file backs, a figure written anywhere other than the generated table, and a reference to a run that does not exist.
Developed in the private engine repository and not yet in the public tree.
The harness below is developed in the engine repository, which is private, and is published from there. Where a file has not reached the public tree yet it is named by path and carries no link: a link into a repository nobody can open is a 404 with a blue underline. pdfluent/pdfluent-sdk
The figures themselves are not on this page. Each one is registered under the identifier above with its source, its run and the date it was measured, and the two that describe our own conversion on the tuning set are held back until the holdout has been run.
A25 · A26 · A27 · A28