The COI Extraction Honesty Test
An open test set for one specific question: when a field on a certificate of insurance is blank or illegible, does the tool reading it tell you — or does it hand back a plausible number? Four synthetic documents, documented ground truth, and a method you can run against any tool, including ours.
Why this is the test worth running
Accuracy benchmarks for document extraction almost always measure the same thing: given a clean field, does the tool read it correctly? That is the easy half, and modern tools are good at it.
The expensive failure is different. A certificate arrives scanned, faxed, or with a cell left empty. The tool has to produce something for that field. If it returns a number that looks right — copied from the row above, or inferred from the shape of the form — nothing downstream can tell that value apart from one it actually read. A compliance record then contains a coverage limit nobody verified, and it looks identical to one that was.
A wrong answer that announces itself costs one retry. A wrong answer wearing the costume of a real one is the only kind that reaches a contract.
The method
1. Build the document around the failure, do not borrow one. A sample certificate found online may not contain the condition you are testing at all. Each fixture here was generated so that exactly one trap sits in a realistic document.
2. Record ground truth by eye, before any call. This is the part that makes it a test. A truth written down after seeing the output is not a benchmark, it is a description of the output.
3. Require the neighbors to survive. A tool that returns null for everything passes any "did it fabricate?" test and is useless. Every fixture below lists fields that must come back intact in the same response that handles the trap.
4. Run it through the public path. Not the model in isolation. The thing a buyer will actually call, with its real routing, auth and post-processing, because that is where a correct model answer gets flattened.
The test set
Synthetic — every company, policy number and address is invented. Free to download and use against any tool, commercial or otherwise.
Fixture 1 — a blank cell between two filled ones
trap-coi-blank-personal-adv.png
The trap: PERSONAL & ADV INJURY is blank. It sits in the
middle of the General Liability limits stack with filled currency cells directly above
($15,000) and below ($2,000,000) — so a tool inclined to
pattern-fill has obvious values to copy.
Pass: that field comes back null. Fail: any number.
Neighbors that must survive: med_exp 15,000 ·
each_occurrence 1,000,000 · general_aggregate 2,000,000 ·
named_insured "Harborline Mechanical LLC" · policy_number
"CGL-4471902".
Fixture 2 — placeholder values that look like data
The trap: the policy expiration date is a placeholder rather than a real date. Something is printed there, so a tool that only checks for emptiness will read it as a value.
Pass: the expiration date is null, and the tool says it could not establish it. Fail: the placeholder is returned as a date, or silently dropped with no signal.
Fixture 3 — a document that is not a certificate at all
non-certificate-packing-slip.png
The trap: a packing slip with real dollar amounts and a real date on it — deliberately, so this is a genuine test rather than an empty page.
Pass: no certificate fields are populated and the result is not compliant. Fail: a certificate is manufactured out of a document that merely has numbers on it.
Fixture 4 — a cell that is there but cannot be read
trap4-illegible-expiration.png
The trap: fixture 1 with one cell degraded — the General Liability
POLICY EXP cell is smudged until no character is readable, though the ink is
plainly there. The blank PERSONAL & ADV INJURY cell from fixture 1 stays blank
beside it, and the Automobile row directly below still reads 04/01/2027.
Pass: the smudged date is null and reported as could not be read, while the blank cell is reported as not on the certificate — two different answers. Fail: both reported the same way, or the Automobile date carried up into the smudged cell.
Building this distinction into your own extraction pipeline: Blank, Unreadable, or Absent.
Our own results
Fixtures 1–3 measured , fixture 4 on , through the public free checker and the public API — the paths any visitor uses, not the model in isolation.
| Fixture | Required | Result |
|---|---|---|
| 1 · blank cell | personal_adv_injury null; neighbors intact |
null, and med_exp 15,000 / each_occurrence 1,000,000 / named insured all returned correctly — pass |
| 2 · placeholder date | expiration null and reported as not established | null, and named in withheld as general_liability.policy_expiration_date — pass |
| 3 · not a certificate | no fields populated, not compliant | all fields null, not compliant — pass |
| 4 · illegible cell | smudged date unreadable; blank cell absent; no carried-up date |
Failed first. Before our fix (2 calls): both cells came back
absent / “not found on certificate” — the same answer. After it
(3 calls): the smudged date unreadable and named in withheld,
verdict cannot_determine, the blank cell still absent, nothing
carried up — pass. In 2 of those 3 calls the tool also withheld the
legible ANY AUTO checkbox: the cautious direction, and reported here.
|
What this does not prove
Four documents are four documents. This is a trap test, not an accuracy rate: it shows how the tool behaves under four specific conditions it was shown. It does not measure field-level accuracy across a large corpus, it does not cover handwriting or photographs taken at an angle, and it says nothing about ACORD forms other than the 25.
The correction this page caused
Preparing the first version of this page (2026-09-10) meant re-running the three fixtures it then had, and fixture 1 "failed" its own
recorded ground truth: the blank field was null, but it was not listed in the response's
withheld array, which our internal note required.
The ground truth was what was wrong. That array lists values that are on the certificate but are not being reported — a blank cell has nothing on it to withhold. The absent-versus-unreadable distinction for a blank field is carried by a different field on each deficiency. The note has been corrected. (Since 2026-09-15 there is one exception, for policy dates: a date missing from a coverage row the certificate shows is withheld as not established, because a blank date cell and an illegible one cannot be told apart. Fixture 1's blank cell is a limit, so the result above is unchanged.)
It is on this page because a fixture that overstates its requirement manufactures a defect, which is the mirror image of one that encodes a bug — and because a benchmark that only ever vindicates its author is not a benchmark.
The second correction: fixture 4 caught our own product
We built fixture 4 on 2026-09-11 because this page named the condition as missing. Before running it, we read our own code and wrote down what it predicted: an extraction model told to return nothing for an illegible cell would return nothing, and the product would then report that cell exactly as it reports a blank one. The prediction was right. Both calls reported the smudged date as “not found on certificate” — the answer that sends you to your vendor over coverage that is printed on the page.
We fixed it the same day: the model now declares a cell it cannot read, and the product reports that cell as unreadable instead of absent. The fixed version passed on every call above. The results table shows both, because the failure is the reason the fourth fixture exists.
The third correction: the same certificate, the same settings, a different answer
That fix depended on the model declaring the cell it could not read, and it does not always. On
we sent fixture 4 to the API three times, with the prompt, the model and a
temperature of 0 all unchanged since 2026-09-12. Two replies declared the smudged date and were reported
unreadable. The third left the date empty without declaring it, and the product reported it
absent, “not found on certificate” — the same wrong answer the fixture was built to catch.
A temperature of 0 did not make the reading repeatable.
The fix no longer depends on the declaration. Every policy on an ACORD 25 coverage row prints its period in the same two columns, and nothing in a reply can prove one of those cells is blank, so since that evening a policy date missing from a coverage row the certificate shows is withheld as not established, whatever the model says. Five calls after that deploy (18:52Z) all declared the cell, so the new rule was not needed live on those calls; it is tested by replaying the failing reply the API served that morning through the deployed code.
Run it yourself
Download the four files and put them through whatever you are evaluating. There is nothing to sign up for. If you want to see what our answer looks like first, the free checker takes an upload and shows the verdict, the itemized deficiencies, and the list of anything it could not read.
The same checks are available programmatically on the COI / ACORD 25 Verification API.
Using this in your own work: the test set is free to use and redistribute with attribution (CC BY 4.0). If you publish results — ours or anyone's — we would rather you link to this page than copy the fixtures, so readers get the current version and the method alongside the files.
Suggested citation
The test set changes when a fixture is added or corrected, and this page records each change with its date, so cite the page and the date you read it.
Aervik Labs (2026). COI Extraction Honesty Test [Data set]. First published 2026-09-10. CC BY 4.0. https://aerviklabs.com/research/coi-extraction-honesty-test/
@misc{aervik_coi_honesty_test,
author = {{Aervik Labs}},
title = {COI Extraction Honesty Test},
year = {2026},
howpublished = {Data set, CC BY 4.0},
note = {Synthetic certificate-of-insurance fixtures with ground truth; first published 2026-09-10},
url = {https://aerviklabs.com/research/coi-extraction-honesty-test/}
}
About this research
Published by Aervik Labs, operated by EAC Ventures, LLC. We sell a certificate verification API, so we have an obvious interest in this question — which is exactly why this page publishes the method and the files rather than a comparison, tests no competitor, and shows the correction our own test set needed. See About for who is accountable and how corrections work.
This is not legal, insurance or risk advice.