90%+

Expert Verdicts Matched

Dozens

Issues Experts Missed

~$2

Cost Per Report

The AI Reviewer That Caught What the Experts Missed

How a global professional services firm put a quality floor under thousands of reviewer hours a month

Across a set of test reports, the system matched over 90% of the issue classes the firm's expert reviewers flag, then surfaced dozens more problems nobody had caught: wrong jurisdictions, figures that contradicted the source cited beside them, entity names that drifted between sections, links that had been truncated and never clicked. Those reports run anywhere from a few pages to a few hundred. A full pass over one costs a couple of dollars in model tokens and takes a few minutes.

Nobody set out to prove that. The brief was to save reviewer time.

Every report the firm sends to a client gets read by a human first. A senior reviewer checks claims against the underlying sources, keeps names and entities consistent from the executive summary through the appendix, and flags anything that could embarrass the client. Reports run from a few pages to a few hundred, so a single review takes anywhere from under an hour to the better part of two days. Across a month that is thousands of reviewer hours, all of it inside a turnaround window of a day or two.

The second problem was the one leadership actually worried about. Reviewer skill varies. Two experienced people reading the same report will not flag the same things, and the gap between them stays invisible until something reaches a client. The firm wanted a floor under quality that did not depend on who picked up the file.

What We Built

We built a review layer that reads a finished report the way a senior reviewer would, then writes its findings straight into the document.

It is not one model with a very long prompt. It is a set of narrow agents, each responsible for a single category of mistake. One hunts contradictions between sections, which slip through because sections get drafted and read independently. One enforces naming: an entity has to be defined on first reference, and every later alias, transliteration, accented spelling, or trading name has to resolve back to that definition. One walks the footnotes, fetches what it can, and reports which sources resolved and which links are dead. Another checks the executive summary against the body it claims to summarize.

Findings come back as inline Word comments and tracked changes, so a reviewer accepts or rejects each one the same way they would handle edits from a colleague. Nothing gets applied silently, and every change carries an audit trail.

Two decisions did most of the heavy lifting. The first was an adversarial pass: before a suggestion reaches a human, a second agent argues against it, and the weak ones get dropped. Comment volume was never the target. An early run left close to a hundred comments on a single report, which is worse than leaving none, because a reviewer wading through that much noise has already lost the time the tool was meant to give back. The second decision was to stop asking a language model to do arithmetic and string matching. Anything a deterministic script can verify gets verified by a script, and the model is left to make the judgment calls that actually need judgment.

What that adds up to is a multi-agent architecture, and it is genuinely complex. Specialized agents run over the same document in parallel, each with its own evaluation harness and its own definition of what counts as a miss. Their findings meet an adversarial pass that argues suggestions down, and a layer of deterministic checks that never guesses. Building it this way is considerably harder than writing one long prompt, and it is the reason the output holds up next to a senior reviewer: every agent can be measured, corrected, and replaced without disturbing the others.

How We Measured It

The firm gave us something most clients cannot: answer keys. For a set of real reports we had both the raw draft and the version a senior reviewer had already marked up. We tuned the agents on one document and tested against the others, keeping the marked-up versions in a folder the model never saw. That turned an argument about whether the output felt good into a measurement.

On coverage, the agents matched over 90% of the issue classes the reviewers flag. Comment-for-comment is a stricter test, and there they landed on the same spot as the human reviewer roughly four times in five on average, and above nine in ten on the most-reviewed reports. Both figures were scored against the reviewers' own markup rather than by the AI itself.

The coverage figure is the one that matters. The agents match the firm's reviewers on nearly every category those reviewers look for, and then keep going, surfacing dozens of issues across the test set that no human had caught. On the measure the firm actually cares about, how much reaches a client that should not have, the system is finding more than its expert reviewers do, and it holds that standard on every report rather than on the ones that happened to get a careful morning.

The Economics

A full pass costs a couple of dollars in model tokens per report, on true per-run accounting from the API rather than a projection. The client had set a ceiling several times higher as the point where the math stopped being obvious, which leaves room to add agents without reopening the business case.

The proof of concept measured cost and accuracy, not hours recovered, so what follows is arithmetic on the figures above rather than something the firm has clocked. Take a report at a few hours of review, well inside their range. The pass runs in minutes for a couple of dollars, beside the reviewer rather than in front of them. If it takes even a quarter off the reading, each report trades a couple of dollars in tokens for about an hour of senior reviewer time, which is hundreds of dollars of expert review at their rates.

Thousands of reviewer hours a month at a few hours each works out to a couple of hundred reports a week. On that volume the trade is hundreds of reviewer hours returned per week for a few hundred dollars, with the machine time running in parallel and nothing added to the turnaround. The quarter is the assumption worth arguing with. Every other figure in those two sentences was measured.

We benchmarked three frontier models against the same sealed answer key. The cheapest one was also the most accurate, measured inside our own harness on this task rather than as a general ranking.

A model with a lower price per token still costs more per job if it burns twice as many tokens getting there, and raw capability does not automatically transfer to a narrow task tuned around a specific harness. We ran at temperature zero so the same document produces the same review twice, which mattered more here than a point of headline accuracy: a reviewer cannot audit a tool that answers differently on Tuesday than it did on Monday.

Why It Worked

• Narrow agents beat one big prompt. Each has one job, so you can evaluate it, fix it, and swap it without touching the rest.
• Signal over volume. The adversarial pass exists to throw suggestions away. A reviewer who stops trusting the comments stops reading them.
• Code where code belongs. Deterministic checks handle the mechanical work, which takes the failure rate on those checks close to zero.
• Graded against the client's own experts, not a vendor benchmark. That is what makes these numbers defensible in a room full of skeptics.

Where It Stands

The proof of concept ran a few weeks from kickoff to handoff. Every agent was built and evaluated individually, and the codebase was documented and transferred to the client's own engineering team so they can keep iterating without us.

The firm is now deciding how to get it in front of reviewers. Our recommendation was to run it silently in parallel with the humans first and compare the two outputs without disrupting anyone's day. It is the only version of the rollout where you find out what the tool really does before anybody starts depending on it.

This case study has been anonymized. The client, their sector, and the subject matter of the reports have been withheld, and the figures have been rounded to protect the client. Outcomes are accurate to the engagement and taken from the proof of concept evaluation.