How to evaluate document AI accuracy: a buyer's method

Vendor accuracy numbers cannot be compared: each is measured on a different dataset, different fields, and a different definition of correct. The only accuracy comparison that holds is one you run yourself, with 100 or more of your own production documents, the schema you actually need, and every candidate scored per field. This guide walks through that method step by step.

Why can't you compare document AI accuracy benchmarks?

Full disclosure: we build bem, one of the platforms you might score this way. The method favors whoever survives contact with your documents.

The benchmarks don't compare because no two vendors measure the same thing. Hyperscience advertises accuracy rates of 99.5 percent. ABBYY Vantage promises "90% accuracy at the start". Reducto claims 99.24 percent parse accuracy. All three were live in September 2026, and none are comparable: different products (end-to-end intelligent document processing, pretrained extraction skills, parsing), unknown datasets, unstated definitions of correct. Character-level OCR, field-level extraction, or documents with zero errors? The pages do not say.

Reducto deserves credit for the exception. It publishes a benchmark hub at llms.reducto.ai, including RD-TableBench for table extraction, so you can at least inspect what was measured. Most vendors publish nothing: no dataset, no field list, just a percentage.

Even an honest benchmark was not run on your scans, your line-item schema, or your definition of a match. The accuracy you will get only shows up in an evaluation you run yourself, and the rest of this piece is that evaluation.

What does 98 percent accuracy really mean on a ten-field document?

Less than you think, because errors compound per field. At 98 percent per field, the chance a ten-field document comes back fully correct is 0.98 multiplied by itself ten times: roughly 82 percent. Nearly one in five documents has at least one wrong field.

On a thirty-field document, a bill of lading or a claims packet, the same 98 percent compounds to roughly 55 percent. Half your documents contain an error. Run the math backwards and it gets stricter: 95 percent document-level accuracy on thirty fields requires roughly 99.8 percent on every field.

This is why a single blended accuracy number is the wrong question to ask any vendor, including us. A platform can score 99.9 percent on invoice numbers, 70 percent on line-item quantities, and quote a flattering blend that hides exactly the fields that will hurt you. Score per field, and weight fields by what a mistake costs: a wrong invoice total is not the same defect as a wrong postal code.

How do you build a test set for a document AI evaluation?

Pull at least 100 documents from your own production traffic, and include the ugly ones deliberately. Every vendor demo runs on clean PDFs. Run yours on the worst fax your ops team received this quarter, the phone photo of a crumpled delivery receipt, the 80-page attachment holding three different documents, the Spanish-language variant. Match your real mix roughly, then oversample the tail; that is where platforms separate.

Next, define the schema you actually need downstream, not whatever the vendor demo extracts. Write out the exact fields your TMS, claims system, or ERP ingests, with types, including the awkward ones: line items as nested arrays, amounts with units. If your system needs a carrier SCAC code, the evaluation tests SCAC codes.

Then label ground truth once, by hand: a human records the correct value for every field, and a second person resolves disputes. It is tedious and the best money in the exercise: the labeled set outlives the evaluation. You will reuse it on new vendors, new versions, and drift checks for years.

How should you score each field: exact match or semantic match?

Run every candidate on the same documents against the same schema, then score per field with typed comparisons, decided before you look at any output.

Exact match after type normalization is the default. Dates compare as dates, so 2026-03-04 and 03/04/26 match. Amounts compare as numbers, so 1,250.00 and 1250 match. Identifiers like invoice numbers and VINs compare exactly after trimming whitespace and case. Without normalization you punish formatting instead of measuring accuracy; with sloppy normalization you forgive real errors. Write the rules down and apply them identically to every candidate.

Semantic match is for free-text fields only: descriptions, addresses, names. Decide what counts, for example "same street address despite abbreviation differences", and hold the line. If a language model judges semantic fields, spot-check its verdicts by hand.

Score wrong and missing separately. A missing value fails loudly; a confidently wrong value flows downstream. A platform that returns null when uncertain behaves better than one that guesses. Keep results per field, per document, per candidate; aggregates come last.

Does a confidence score of 0.95 actually mean 95 percent?

Often not. This is the most revealing test you will run. If you plan to route low-confidence fields to human review, the confidence score is the routing signal, and an uncalibrated signal routes the wrong things.

Bucket every extracted field by reported confidence and measure realized accuracy per bucket against ground truth. Calibrated means realized accuracy lands within a few points of reported, in every bucket. The common failure is a platform that stamps nearly everything 0.97 regardless of difficulty. That score is decoration, and you cannot route on decoration.

Then count routing behavior at the threshold you would run. Two numbers matter. The review rate: what fraction of fields go to humans, your staffing cost. And escaped errors: wrong values above the threshold, your incidents. Check source grounding too: a reviewer who can jump to the cited spot clears a routed field in seconds. A candidate at 96 percent per field with honest confidence scores beats one at 98 percent that fails silently. The first turns its errors into review tasks. The second ships them.

How do you test determinism and drift in document AI?

Pick five documents and run each through every candidate five times. Count fields that change value or confidence between runs. The right answer is zero. Any variance means rerunning is a gamble, reprocessing after an outage can rewrite historical records, and in a regulated workflow, different answers on identical input is an audit finding, not a quirk.

Drift is the slower version of the same problem. Rerun the full set a month later, or right after the vendor announces a model update, and diff every field. Hosted models change underneath their APIs, silently. Ask each vendor: are function versions pinned, do we control when an upgrade applies, do you run regression tests between versions, and can outputs be replayed against the schema version that produced them? Vague answers here predict the incident where last quarter's documents reprocess differently.

Keep the labeled set and rerun it quarterly; the evaluation becomes a regression harness, not a one-time procurement gate.

What does a document AI evaluation scorecard look like?

One row per metric, one column per candidate. This is the version we would want forwarded to us.

Metric	How to measure it	What good looks like
Per-field accuracy	Fields matching ground truth after typed normalization	No critical field below your floor; high nineties where money moves
Document-level accuracy	Share of documents with zero field errors	Consistent with the compounding math for your field count
Confidence calibration	Realized accuracy per reported-confidence bucket	Within a few points of reported, every bucket
Escaped errors	Wrong values above threshold, per 1,000 fields	Near zero on money and identifier fields
Review rate	Share of fields routed to review	Low enough for your ops team to staff
Determinism	Identical fields across five runs of a document	100 percent
Drift	Fields changed on rerun after a month or a version change	Zero without a deliberate version change
Cost per document	API price on your page mix plus review minutes at loaded cost	Priced per correctly processed document, not per API call

Price the evaluation per document, all-in: each candidate's charges on your real page mix, plus the review labor its routing generates, plus retries. A cheap platform with a 30 percent review rate is not cheap. Divide total cost by correctly processed documents and the ranking often flips.

We will be scored on this table too, so here is where bem stands. bem automates entire business processes end to end: it turns unstructured data into auditable outputs, then makes and records the decisions in between. Every value is confidence-scored per field, every function version behind the V3 API gets automatic regression testing and drift detection, and every output is replayable against the schema that produced it: same input, same output until you publish a new version. Run your hundred documents through us and check.

The vendor that encourages this evaluation is telling you something. So is the one that resists it.