Paper Trail — Adversarial Evals for Document AI 2026

Static evals of document AI measure a fixed test set, so they only report how a model does on the errors someone already thought to write down. Paper Trail runs a deterministic simulated economy instead — contracts, invoices and bank statements with errors injected from a tagged taxonomy — and puts an adaptive adversary in the loop that reweights injections toward whatever the auditor misses, then lets an LLM propose entirely new error types.

Each round the simulator generates honest economic activity, the adversary picks which errors to inject and at what difficulty, and an agentic LLM auditor investigates the rendered documents through four read-only tools before emitting structured findings. A scorer matches findings against a canonical ground-truth event log; the adversary updates a Thompson-sampling posterior over 24 arms, rewarded when the auditor misses.

Across 25 audited rounds the adversary never found a recall blind spot. The adaptive arm scored 0.959 mean recall against 0.960 for the static control on an identical seed — no measurable effect, and no error type collapsed.

• No reward signal: the auditor missed 4 of 94 scored injections. Thompson sampling over 24 arms cannot concentrate on a ~4% base rate — final posterior means span 0.12–0.29, so sampling stays effectively uniform.
• The exploration floor eats the budget: 24 arms forced once every 5 rounds is 4.8 mandatory slots out of 10 per round — ~48% of every plan is exploration by construction.
• Wrong axis: reward is 1 iff the auditor missed. Recall is saturated at ~0.96 while precision is the real weakness — 0.658 mean, 1–10 false alarms per round. A miss-based objective is structurally blind to it.
• LLM-proposed errors didn't help: two novel templates passed the schema and both were caught every time they were injected — neither beat any hand-built type.

Rounds 10 and 13 recorded recall 0.00 with zero findings — exactly the headline this project was built to find, and false. The tell was that both rounds also missed error types the auditor catches reliably. Re-running round 10 against its saved documents returned 24 findings where the original returned 0.

Root cause: the final answer must carry every finding in one JSON object and max_tokens_per_turn was 4096. A round with many findings truncated the JSON mid-object, the retry truncated identically, salvage returned an empty list, and the scorer faithfully recorded recall 0.00 — which the adversary then learned from. Uncorrected, the data claimed an LLM-invented error type defeated the entire hand-built taxonomy (recall 0.55, lowest of any type); excluding the two broken rounds it is 1.00.

Fixed by retrying the final answer with a 16k ceiling on stop_reason == "max_tokens", persisting raw text whenever a report parses empty, and three regression tests. An earlier bug of the same shape: ground truth tags transaction ids, but transactions have no rendered document — so every payment-side error was unmatchable regardless of auditor performance.