How to benchmark OCR models on your own documents: a 200-page gold set, a fair harness, and the metrics that map to money
TL;DR
The only benchmark that predicts your extraction accuracy is one run on your own pages. Public leaderboards tell you which models are worth testing; they do not tell you how many of your invoices will need a human.
Build a 200-page gold set stratified by document family, difficulty and language. Below about a hundred pages, the differences you see between models are mostly noise.
Label from the page image, never from the ERP. Double-label one page in five and record the agreement rate — it is the ceiling any model can reach, and it belongs in the report.
Report per-field accuracy and the document-level straight-through rate per family, not one blended number. Straight-through rate sets the size of the human review queue and therefore the real cost per page.
Resolution mismatch is the single most common reason a local model “loses” to an API in an internal test. Same image, same prompt, same normalizer, same retry policy, for every model.
Questions people ask
How do I benchmark OCR models on my own documents?
Write a schema per document family with fields, types and match tolerances. Assemble 200 stratified pages from at least three months, split into a tuning half and a held-out half, and label them from the image. Shortlist three to five models including a Tesseract baseline. Build a harness that gives every model the same input and scores every output the same way, then decide by straight-through rate per family.
How many pages do I need to benchmark a document model?
Two hundred, stratified by family, difficulty and language, is the floor. It is the smallest set that gives a usable per-family read and can still be labeled by one person in a few days. Below about a hundred pages the differences between models are mostly noise, and twenty pages cannot separate two models at all — the gap between 17 and 18 correct is one page. Larger is better once the harness works.
What is straight-through rate in document processing?
The straight-through rate is the share of documents where every required field is correct within its tolerance, so the document can be posted without a person touching it. It is the most useful number in a document AI benchmark because it sets the size of the human review queue. Report it per family; a blended figure is dominated by your easiest template. Pair it with the human-touch rate at your confidence threshold.
Why do public OCR leaderboards not predict accuracy on my documents?
Four reasons. Your pages are dominated by a handful of supplier templates plus a long tail, while public sets are deliberately diverse. Latvian diacritics, Norwegian ø, German ß and mixed-language pages are thinly represented. Faxed, stamped and phone-photographed pages are the failure mass in real queues and a minority in public sets. And popular public datasets end up in training data, so a score on them is not a prediction.
How do I compare a local model with OpenAI or Anthropic fairly?
Send the same rasterized page at the same resolution to every model, with the same prompt and schema byte for byte, the same normalizer and the same retry policy. Run each model at least twice and report the mean. Record the model identifier and date, because API models drift between quarters. Price API cost from tokens at list price and local cost from measured GPU-hours, and record whether the pages left the EU.
What mistakes invalidate an OCR benchmark?
Tuning the prompt while looking at held-out scores, which turns the test set into a training set. Reporting the best of five runs or only the family where your preferred model won. Feeding models different image resolutions. Penalizing “01.10.2026” against “2026-10-01”, which is a normalizer bug reported as a model error. Taking ground truth from the ERP instead of the page. Sets under 200 pages. A benchmark you cannot re-run in an hour.
Want this worked out on your documents?
We will price a real sample against OpenAI or Anthropic and tell you whether Local document models or a local model on your hardware is the cheaper first step.
The only benchmark that predicts your extraction accuracy is one run on your own pages. Build a 200-page gold set stratified by document family, difficulty and language, label it against your schema, and score every candidate — local or API — on identical inputs with identical normalization. Report per-field accuracy and the document-level straight-through rate, not one blended number. Public leaderboards tell you which models are worth testing; they do not tell you how many of your invoices will need a human. A team with the pages and a spreadsheet can have a defensible result in a working week, and the same harness re-runs in an hour when the next model ships.
This is the method we use inside Ækora engagements before a model is chosen, and the one we recommend to any European buyer comparing on-prem models with cloud APIs. Nothing in it needs a research budget. It needs a schema, a labeler, and discipline about what you do not look at.
Why public leaderboards mislead for your pages
Leaderboards for OCR and document understanding are built on public datasets: academic papers, English forms, receipts photographed for a competition, tables cut from annual reports. Four ways that differs from an accounts-payable inbox in Riga or a claims queue in Munich:
Distribution. Your pages are dominated by a handful of supplier templates plus a long tail. Public sets are deliberately diverse. A model that shines on diversity can be beaten on your top template by a smaller one.
Language and script. Latvian diacritics, Norwegian ø, German ß and mixed-language pages are thinly represented. Leaderboard rank says little about how a model reads them.
Scan quality. Faxed, stamped, stapled and phone-photographed pages are the failure mass in real queues and a minority in public sets.
Contamination. Popular public datasets end up in training data. A score on a set the model may have seen is not a prediction for a set it has not.
Leaderboard order and your order disagree often enough that using one to pick a model is a coin flip you paid for. Use leaderboards to build a shortlist — our local OCR model comparison is one — and your own pages to choose. If you are weighing a hosted OCR plan, the open weight shortlist worth benchmarking is costed against Nanonets.
Build a 200-page gold set
Two hundred pages is the smallest set that gives a usable per-family read and is still labelable by one person in a few days. Larger is better once the harness works; below about a hundred, the differences you see between models are mostly noise.
Stratify; do not sample at random
A random 200 from the inbox over-represents the easy majority. Stratify on three axes so every cell that matters has enough pages to say something:
Example stratification plan for a 200-page gold set at a mid-sized European accounts-payable and claims team. Counts are a template — weight your own cells by monthly volume and by where errors currently cost you. The non-English column counts pages already included in the other three.
Document family
Clean digital
Scanned / stamped
Photographed / handwritten
Non-English or mixed (within)
Total
Supplier invoices
30
25
10
15
65
Receipts and expense claims
5
10
20
10
35
Contracts and agreements (sampled pages)
20
10
0
10
30
Forms and applications
10
15
15
15
40
Long tail (credit notes, statements, letters)
10
15
5
10
30
Total
75
75
50
60
200
Example stratification plan for a 200-page gold set at a mid-sized European accounts-payable and claims team. Counts are a template — weight your own cells by monthly volume and by where errors currently cost you. The non-English column counts pages already included in the other three.
Pull pages from at least three months so one bad scanner week does not define “hard”. Include the templates you process most and the ones that currently generate the most tickets. Keep the source file, the page image at a fixed resolution, and the family and difficulty tags together in one manifest. If claims are one of your families, build its cells from what a claims file actually contains rather than from the claim form alone.
Label against a schema, then double-label a subset
The schema comes before labeling. For each family, list the fields, their types, whether they are required, and the tolerance for a match: exact for an IBAN, normalized for a date or an amount, fuzzy for a free-text description. Then have one person fill in ground truth for every page from the image — not from the ERP, which contains its own errors. Have a second person label a subset independently, one page in five, and compare. Where the two disagree, the field definition is ambiguous, not the labeler: fix the definition, adjudicate, and record the agreement rate. That rate is the ceiling any model can reach, and it belongs in the report.
Freeze the set, version it, and keep a held-out half that nobody uses for prompt tuning. If a vendor asks for sample pages to configure their product, give them pages from the tuning half only.
Metrics that map to money
Definition · Straight-through rate
The straight-through rate is the share of documents where every required field is correct within its tolerance, so the document can be posted without a person touching it. It is the single most useful number in a document AI benchmark, because it sets the size of the human review queue and therefore the real cost per page.
One accuracy number hides everything you need to know. Score at three levels and add operational measures:
Field-level exact match. Prediction equals ground truth character for character. Right for identifiers: VAT numbers, IBANs, invoice numbers.
Field-level normalized match. Both sides are canonicalized first — dates to ISO 8601, amounts to integer cents with a currency, whitespace and case folded, “1.234,56” and “1234.56” made equal. Right for dates, totals and names. Most “errors” in naive scoring are normalization gaps.
Precision and recall per field and per line item. For repeating fields such as invoice lines, match predicted rows to gold rows by a key — line number, or description similarity plus amount — then count matched, missing and invented rows. A model that adds a line that does not exist has a precision problem that a match rate will not show.
Document-level straight-through rate, as defined above, reported per family. A blended figure is dominated by your easiest template.
Human-touch rate. The share of documents your confidence threshold sends to review — larger than the failure rate, because the threshold also catches correct pages the model is unsure about. This is the number that becomes headcount.
Latency. Median and 95th percentile per page, end to end including preprocessing, because the 95th percentile is what a same-day SLA feels.
Cost per page. Tokens × list price for an API; GPU-hours × hourly rate ÷ pages for a local model. Record it beside accuracy: a model that clears a few more documents straight through can be worth several times its compute.
The harness: five components
A benchmark harness is a pipeline with one job: give every model the same input and score every output the same way. Five components, each replaceable:
Structure of a document-model benchmark harness. Each component is a small module; the interfaces between them — a manifest, a raw output record, a normalized record, a score table — are what make results comparable across runs and months.
Component
Job
Input
Output
Rule that keeps it fair
Loader
Reads the manifest, rasterizes PDFs at a fixed DPI, attaches gold labels and tags
Gold set + manifest
Page image + gold record
One resolution per family, applied to every model
Model adapter
Sends page, prompt and schema to one model; records raw output, tokens, latency and version
Page image + prompt
Raw output record
Same prompt and schema text for all; temperature 0; one adapter per model or API
Normalizer
Parses JSON; coerces dates, amounts, identifiers and text to canonical forms
Raw output and gold record
Normalized prediction and gold
Applied identically to predictions and ground truth
Scorer
Field match, line-item matching, per-document straight-through, confidence thresholds
Normalized pairs
Score table per page × field
Tolerances come from the schema, not from the scorer
Report
Aggregates by family, difficulty, language and model; adds latency and cost
Score table
Results table + failure samples
Records model version, prompt hash, date and set version
Structure of a document-model benchmark harness. Each component is a small module; the interfaces between them — a manifest, a raw output record, a normalized record, a score table — are what make results comparable across runs and months.
The adapter is where local and API models meet. A local adapter calls a serving endpoint on your GPU; an API adapter calls OpenAI, Anthropic or Mistral. Everything upstream and downstream is shared, which is the point: the comparison is between models, not between two pipelines that happen to contain models.
Comparing local models and APIs fairly
Most local-versus-cloud comparisons we see are unfair in one direction or the other without anyone intending it. The checklist:
Same image. The same rasterized page at the same resolution goes to every model. Sending a high-detail tile set to the API and a downscaled image to the local model — or the reverse — is the most common silent bias.
Same prompt and schema, byte for byte. If one model needs a hint to return valid JSON, give the hint to all.
Same normalizer. Never let a model’s own output format decide whether it matched.
Same retry policy. One attempt each, or the same bounded retry on invalid JSON for all. Count retries in latency and cost.
Same number of runs. Run each model at least twice and report the mean. If two runs differ materially, the model is unstable at that setting, and that is itself a result.
Pinned versions. A local model’s weights are a file with a hash. An API model is whatever the vendor serves that day. Record the model identifier and the date, and expect API results to drift between quarters.
Cost on the same basis. API cost from tokens at list prices at the time of writing; local cost from measured GPU-hours at your rental or amortized rate. The figures in local LLM vs OpenAI cost and GPU sizing for document processing are the planning defaults; your harness replaces them with measurements.
One asymmetry is legitimate and should be recorded rather than hidden: where the page went. A local run keeps every page inside your infrastructure; an API run transfers it. For GDPR-relevant families that fact belongs in the results table next to accuracy, because it constrains which winners you are allowed to deploy — the European buyer’s guide covers the transfer questions in detail.
A results template
The report is one table per document family, one row per model. Fill it only with what the harness produced.
Results template for one document family. Every cell is a placeholder — fill it from your own harness output. Never carry a number from a vendor page or a leaderboard into this table. Add one table per family and a summary row weighted by monthly volume.
Model (version, date)
Field exact match
Field normalized match
Line-item precision / recall
Straight-through rate
Human-touch rate at threshold
p95 latency per page
Cost per page
Pages left the EU?
Tesseract + rules (baseline)
your value
your value
your value
your value
your value
your value
your value
No
Compact document VLM, local
your value
your value
your value
your value
your value
your value
your value
No
General VLM, local
your value
your value
your value
your value
your value
your value
your value
No
Frontier API model
your value
your value
your value
your value
your value
your value
your value
Yes — record the region
Labeler agreement (ceiling)
your value
your value
your value
your value
—
—
—
—
Results template for one document family. Every cell is a placeholder — fill it from your own harness output. Never carry a number from a vendor page or a leaderboard into this table. Add one table per family and a summary row weighted by monthly volume.
Two rows are not optional. The Tesseract baseline tells you whether a family needs a VLM at all. The labeler-agreement row tells you when a model is at the ceiling and further tuning is measuring noise. A model within a point or two of labeler agreement on a family is done; spend the effort on the next family.
Pitfalls that invalidate a benchmark
Leakage. Tuning the prompt while looking at held-out scores. Every prompt iteration on the test half converts it into a training set. Tune on the tuning half; score the held-out half once per model version.
Cherry-picking. Reporting the best of five runs, or only the family where your preferred model won. Report all runs and all families, weighted by volume.
Resolution mismatch. Covered above and worth repeating: it is the single most common reason a local model “loses” to an API in an internal test.
Scoring the model’s own format. Penalizing “2026-10-01” against “01.10.2026”, or “EUR 1,234.56” against “1234.56”, is a normalizer bug reported as a model error.
Ground truth from the ERP. The system of record has its own mistakes and its own post-processing. Label from the page.
Tiny sets. Twenty pages cannot separate two models; the difference between 17 and 18 correct is one page. Two hundred, stratified, is the floor.
Ignoring the queue. A model that is a point more accurate but poorly calibrated can send more pages to review, not fewer. Score the human-touch rate at the threshold you will actually use.
One-time benchmark. Models ship quarterly and APIs change under you. A harness you cannot re-run in an hour is a report, not a benchmark.
The procedure, start to finish
Write the schema per family — fields, types, required flags, match tolerances. Get finance or operations to sign it; they own what “correct” means.
Assemble 200 stratified pages from at least three months, with source file, fixed-resolution image and tags in one manifest. Split into a tuning half and a held-out half.
Label from the image; double-label one page in five; adjudicate; record agreement; freeze and version the set.
Shortlist three to five models — a Tesseract baseline, one compact local VLM, one general local VLM, one API — from the model comparison and your residency constraints.
Build the harness: loader, one adapter per model, a shared normalizer, a scorer that reads tolerances from the schema, and a report that records versions.
Tune prompt and threshold on the tuning half only, with the same prompt text for all models.
Run the held-out half twice per model and fill the results template with means. Add latency, cost per page and where the pages went.
Decide by straight-through rate and human-touch rate per family, subject to residency and hardware, and write the decision down with the set version and date.
Schedule the re-run: quarterly, or whenever a model or API version changes, on the frozen held-out set.
What to do with the result
A benchmark is a decision document. If a compact local model reaches the labeler ceiling on your two biggest families, you have both a model and a hardware class — usually a single 24 GB card — and the API belongs on the exception queue, the hybrid pattern. If nothing clears a family, the finding is a labeling or schema problem more often than a model problem. Either way you now own an asset — a versioned gold set and a harness — that turns every future vendor claim into a one-hour test, which is why our RFP question list asks vendors to run on your pages before they quote.
Measure straight-through rate per family first, human-touch rate at your threshold second, cost per page third. If you would rather have the harness built and run for you, that is the first step of a local document models engagement — and for teams that want the output without the hardware, it is how we scope bulk document processing: the same gold set decides the schema and the validation rules before the first batch runs.