Half the invoices an AI agent wrote were wrong, and nothing said so.
Keyross' yoke checks every document an agent writes, at the moment it writes it, against the official rules and your own data. It doesn't make the agent smarter. It stops wrong invoices from leaving silently. We measured that on 300 runs, graded by three judges that are not Keyross.
What 100 invoices became, per setup
The same agent (Claude Haiku 4.5), the same prompt, the same 50 purchase orders, twice each.
Correct means accepted by all three judges: an independent validator of the official CEN rules, the official Factur-X schema, and a check against the order. The 11 still shipped wrong in the last row are all XML schema violations — the one property the yoke does not check yet.
invoices per 100 shipped with a fatal official-rule violation, with either yoke.
correct invoices with the yoke on the official rules. Exact paired test, Holm-adjusted p = 0.005.
invoices where Keyross and an independent validator of the official rules gave the same verdict.
per check (mean; 26 and 33 ms per run, at 1.7 and 2.1 checks a run). The real overhead is the model writing again: 2 to 3 US cents per invoice.
It does what it claims — and the test found where it stops.
No invoice with a fatal CEN rule left either yoked setup: 0 of 200, against 33 of 100 without. No invoice billing an amount that differs from the order left the setup that checks the order: 0 of 100, against 26 of 100 without.
50 → 64 % with the official rules (Holm-adjusted p = 0.005) and 50 → 60 % with the rules and the order (p = 0.021). Paired, the yoked arm wins 17 blocks and loses 3 (rules), 13 and 3 (rules and order). Within each yoked arm, 13 and 12 runs are rescues: a write refused, then fixed by the agent knowing only which check failed. The rest of the net gain is the first write, sampled independently in each arm.
Keyross runs the CEN EN 16931 validation artefacts unmodified. On 557 of 557 invoices graded, it reported exactly the fatal rules that an independent implementation (easybill's validator, in Docker) reported. Its order checks agree with the benchmark's order judge on 547 of 557; the 10 differences are one half-cent tolerance case, open for calibration, and change no result.
A check takes 15 ms on average (15.2 and 15.6 ms; medians 14.9 and 14.4). The cost is the agent writing again: +80 % and +140 % tokens, +18 s and +31 s, +0.018 and +0.032 USD per invoice at list prices.
The XML schema is not checked yet, so 11 % of invoices still ship with a structure error — stated before the run, and next on the roadmap. And a bare category such as order.lines is often not enough for a small model to find its mistake: 14 % and 29 % of invoices end blocked, for a person.
Agents writing official documents need a check at the write. Instructions are not one.
With clean data and a worked example in its prompt, the agent still shipped a defective invoice 50 times in 100. Its first attempt was right about half the time in every setup (50, 51, 48 %): the yoke does not make it write better, it stops what is wrong.
One invoice in three broke a fatal rule of the standard — a missing delivery country on an intra-EU supply, totals that don't add up, too many decimals. These rules are public and executable; nothing applied them at the write.
One invoice in nine passed every official rule and the schema, and still billed the wrong amount. Only the company's own data — here, the order — could catch it. That is what Keyross' delta checks add to the standard.
A few cents on one line, carried consistently into every total; a VAT amount one cent off, inside the official tolerance; an XML element out of place. A person skimming the invoice could easily miss them; a check at the write does not.
Since 1 September 2026, every VAT-registered company in France must be able to receive e-invoices, and large and mid-sized companies must issue them; small companies and micro-enterprises follow by 1 September 2027 at the latest (DGFiP). The invoices follow the EN 16931 standard — the rules this benchmark ran.
Real lines from invoices the agent shipped without the yoke
| Line | Should be | Written | Off by |
|---|---|---|---|
| 12.75 × 89.95 | 1,146.86 | 1,146.81 | −0.05 |
| 0.5 × 89.95 | 44.98 | 45.00 | +0.02 |
| 5 × 7.45 | 37.25 | 37.50 | +0.25 |
| VAT at 20 % | 263.33 | 263.32 | −0.01 |
Each of these invoices was coherent with itself: the totals followed the wrong line, so the official validator and the schema accepted it. The first error alone shipped in 8 invoices. With the order in its context, the yoke refused every such write.
Two situations hold most failures — both missing from the example it was given.
Correct invoices out of 20 per scenario. The worked example in the prompt is a domestic invoice, with no delivery data and no shipping charge.
| Scenario | Agent alone | + yoke: rules | + yoke: rules and order |
|---|---|---|---|
| Domestic | |||
| Exempt line | |||
| Reverse charge | |||
| Allowances and charges | |||
| Intra-EU supply |
- Intra-EU supply. The invoice must name the country the goods were delivered to (rule BR-IC-12). The agent rarely wrote it where the standard expects it: BR-IC-12 was the most frequent official-rule red flag, 57 and 68 times. Without the yoke, all 20 of these invoices shipped with fatal errors; with it, they were stopped, and 29 of the 43 blocked invoices come from this scenario.
- Shipping charges. When the order had one, the agent placed the charge total out of order in the XML — in 9 of 10 runs, in every setup, identical because no setup checks the schema.
This tells where a check's feedback must get more precise, and which document structures an agent needs to be shown.
Each error disappears exactly where a check covers it.
| Per 100 invoices shipped… | Agent alone | + yoke: rules | + yoke: rules and order |
|---|---|---|---|
| with a fatal official rule | |||
| with an amount that differs from the order | |||
| … of which valid for every official rule and the schema | |||
| with an XML schema violation |
One invoice can have several kinds of error, so the rows overlap. Bars on a 0–100 scale.
| Checked at every write | Agent alone | + yoke: rules | + yoke: rules and order |
|---|---|---|---|
| Content: the official CEN rules | no | yes | yes |
| Amounts: the invoice against its order | no | no | yes |
| Structure: the XML schema | no | no | no — next |
Declared in the protocol before the full run. Where the answer is yes, the error count above is zero; where it is no, errors ship.
What these rates cost at your volume.
The rates below are the ones measured. The costs are yours: change them. The defaults are illustrative assumptions, not measurements.
| Per month | Correct | To a person | Shipped wrong | Total cost | Difference vs agent alone |
|---|
Total cost = model tokens + blocked invoices handled by a person + wrong invoices that ship. Difference = a setup's total cost minus the agent alone's: negative is cheaper. Model costs are measured in USD at list prices (2.5, 4.3 and 5.7 cents per invoice) and counted 1 USD = 1 €; the difference is negligible here. The last row is a projection, not a measurement: it assumes the schema step blocks the 11 % of invoices that now ship with a schema violation, and rescues none of them.
Five hypotheses, five answers — including the unfavourable one.
| Hypothesis | Result | |
|---|---|---|
| H1 | The yoke on the official rules increases correct invoices. | supported 50 → 64 %, p = 0.005 |
| H2 | Risk: the yoke makes the agent bend the order to pass the rules — more valid-but-wrong invoices. | not observed 11 → 11 |
| H3 | Sanity: the first attempt is equally good in every setup. | holds 50, 51, 48 % |
| H4 | The yoke on the rules and the order increases correct invoices. | supported 50 → 60 %, p = 0.021 |
| H5 | Risk: checking the order makes the agent give up more often. | observed 0, 14, 29 % blocked |
p-values: exact two-sided McNemar test on paired runs, Holm-adjusted for the two tested comparisons. Between the two yoked setups there is no difference that 100 pairs can detect (p = 0.39).
Pre-registered, graded by judges that are not Keyross.
- Tasks
- 50 fictitious purchase orders in five scenarios: domestic, exempt line, intra-EU supply, reverse charge, allowances and charges. Each run twice.
- Setups
- The same Deep Agent: alone, with
Yoke(gauge="einvoice"), and with the order in the yoke's context. 300 runs. - Model
- Claude Haiku 4.5, provider defaults, the same prompt with one worked example; at most 4 writes and 12 model calls per run.
- Judges
- easybill's EN 16931 validator (CEN rules 1.3.16), the official Factur-X 1.09 schema, and the match with the order. None of them is Keyross.
- Protocol
- Written and frozen before the first paid run; six dated deviations, each with its reason; two interim looks disclosed; every result published.
- Human review
- 21 delivered invoices drawn at random, 7 per setup, reviewed blind to the setup: the reviewer's verdict matches the three judges on 21 of 21.
The easiest version of the task, on purpose.
- Reading. The agent received a structured order, so the right invoice could be computed and 300 invoices graded. Real inputs are emails and PDFs: harder for the agent, and where a check matters most. The error rates measured here are likely a floor.
- One model, one prompt. A stronger model or a richer example would change how often the agent is blocked.
- By construction. In the setup that checks the order, the yoke and the order judge test the same property with separate code: its zero wrong amounts is expected. What it measures is whether the agent gets there — and it didn't, 29 times in 100.
- Parties, dates, payment details are not compared with the order by any automated judge; the blind human review read them on 21 invoices. It reads content, not XML structure, so it does not test the schema judge.
Each next step is measured with the same benchmark.
- The schema step. Check the XML structure before the official rules. Target: no invoice shipped wrong without anyone being told.
- Sharper feedback, never the answer. Name the line and the field that failed — never the expected value. Target: fewer invoices blocked for a person.
- Messy input. The same 50 orders as emails and PDFs, with the same answer key and the same judges: the case agents are actually used for.
- Supplier invoices. The received PDF checked against the company's own purchase order — the check the order setup already runs.