codebunker.ai builds review tooling for companies that put AI agents in front of customers. The engine scores each conversation from the steps the agent took, with no language model involved, and sets the review threshold from conversations you already know went wrong. You choose the share of failures you can afford to miss. It tells you how many conversations a person has to read, and it writes down why. Tested on 15,436 public agent conversations from fourteen models.
A company running an agent with customers either reads a few percent of conversations by hand or sends all of them to a second model that grades them. The first is cheap and says nothing about the 97% left unread. The second costs about as much as the agent did, and leaves a model grading a model. Neither produces a sentence you could put in front of a regulator about the conversations nobody looked at.
public agent conversations in airline, retail and telecom customer service, fourteen models, outcomes known. This is the data every number on this page comes from.
second-model grading calls on telecom traffic at a 10% miss bound. The conversations not graded come with a written statement about what they can contain.
pooled domains where the observed miss rate at a 10% bound stayed under 10%: 7.3% airline, 9.4% retail, 9.0% telecom.
Lawyers have done something like this since 2011. In document review the software marks most of a corpus irrelevant, a person reads a sample of that pile, and the share of relevant documents found in the sample is reported to the court as the elusion rate. We took the idea, gave the threshold a proof, and pointed it at agents. Data: public τ²-bench trajectories. Everything else, including what went wrong, is in the paper.
Each conversation becomes a short list of what happened: which tools the agent called, in what order, how long between them. For every kind of task, the engine learns from past successful runs what a normal sequence looks like. A new conversation gets a score for how far it strays from that. The threshold comes from conversations you already know failed, through a rule that bounds how many future failures land under it.
A refund that never calls the refund tool. A cancellation that calls it twice. Most failures in the public data were deviations of this kind, and a per-task reference catches them without reading a word of the transcript.
Give it n conversations that failed and a miss rate α. It reviews everything scoring above the ⌊α(n+1)⌋-th lowest of those. The expected share of future failures below the line is at most α. That is a theorem (Angelopoulos, Bates and colleagues, 2024), not a tuning result. Its one condition, that tomorrow's failures resemble today's, is printed on the certificate.
There is no neural network inside. Fitting a reference takes seconds on a laptop and scoring takes 0.36 milliseconds per conversation. The same input always gives the same model and the same decisions, down to the hash, so a second party can rerun it and get the same answer.
One record per step: who acted, what kind of step, which tool, when, how many tokens. The transcript text stays with you.
From your own history, no labels needed. A hash-based split keeps the calibration set apart from the reference set.
Which conversations to read, and a statement about the ones you did not. Every decision goes into a hash-chained log that anyone can verify.
# fit a reference from past sessions POST /v1/models { "sessions": [ … ] } { "model_id": "f59e…", "n_calibration": 1895 }# a review policy with a miss-rate bound POST /v1/models/{id}/triage { "positives": [ … ], "alpha": 0.10 } { "policy_id": "9b1c…", "tau": 4.21, "miss_bound": 0.10 }# which of these does a person have to read? POST /v1/models/{id}/review { "policy_id": "9b1c…", "sessions": [ … ] } { "n": 1895, "n_review": 1137, "decisions": [ … ] }# anyone can check the chain GET /v1/ledger/verify { "ok": true, "records": 3792 }
The only setting. It is a bound on the share of future failures that slip through, not a score on a classifier.
Pooled telecom, 1,895 test conversations, 248 known failures used to set the threshold. Observed miss rate: 9.0%.
Against grading every conversation with a second model. A one-line rule, "did the task's expected tools get called", reaches 56% on the same data. We publish that too.
Everything here comes from the public conversations released with τ²-bench: fourteen models, three customer-service domains, outcomes known. We wrote the pass marks down before running, got the protocol wrong three times, and recorded each error in the paper. The chart is the pooled telecom result.
Share of telecom traffic sent to review at each miss bound. Same calibration rule for every method, 248 known failures. Lower is cheaper. At a 10% bound the engine reviews 60% of traffic. The plain task-conformance rule reviews 44%. A nearest-neighbour detector on generic features reviews 82%.
Observed miss rates at the 10% bound were 9.0% for the engine and 9.5% for the rule. At the 20% bound both came in slightly over, 23.3% and 22.6%. The guarantee is on the expectation, and a single draw can land above it. We report the draw.
| Measurement | Study | Value | Note |
|---|---|---|---|
| Separation of failures from successes (AUROC), per-task reference | A | 0.67–0.90 | 9 model and domain pairs |
| Same, task-conformance rule | A | 0.67–0.90 | one line of code |
| Same, nearest-neighbour baseline | A | 0.52–0.68 | generic features |
| Observed miss at a 10% bound, airline / retail / telecom pools | A | 7.3 / 9.4 / 9.0% | bound held |
| Failures invisible to structure, telecom | A | 3–13% | retail 8–30% |
| Structure of a run in progress predicts a runaway coding run | B | AUROC 0.51 | tokens so far: 0.64. Negative. |
| Decision cost per conversation | bench | 0.36 ms | CPU, no GPU |
| Replay after save and load | bench | bit-exact | model identified by hash |
What did not work. We first set out to stop agent runs early by watching their structure, on 3,665 public coding-agent runs. The structural score predicted nothing that a token counter did not predict better. That study is in the paper as a negative result, and it is the reason this page is about customer-service agents and not about saving tokens.
Hypotheses and pass marks are written before the data is opened. Results that fail are reported with their numbers.
The method, the certificate and its two conditions, the per-task reference, 15,436 public conversations from fourteen models, a negative result on early stopping, and a deployment protocol for a clinical assistant. Thiago Caserta.
A healthcare assistant in Brazil, instrumented to emit events without any patient text. The pass marks are written. The data has not accumulated yet. Results will appear here whichever way they go.
The engine's own proving ground. Twelve of thirteen pre-registered criteria met. The one that failed: drift was detected three times out of three but attributed to the right cause only once. Code and seeds on request.
codebunker.ai was started in 2026 by the people behind codebunker, a software studio in Brazil. We think the expensive part of running AI agents is finding out, afterwards, what they did. That is a measurement problem. We build the instrument and publish how well it works, including when it does not.
Short answers. The long ones are in the paper.
No. There is no neural network in it. It is three statistical operators fitted on your event history, a score, and a calibration rule. It cannot make anything up. It can only miss, at a rate you set.
Given n past failures and a miss rate α, the expected share of future failures scoring under the threshold is at most α. Two conditions: future failures have to resemble the ones you calibrated on, and the reference has to be fitted on data kept apart from the calibration set. Both are stated on the certificate and both can be checked.
Anything that looks normal from the outside. A polite, confident, wrong answer delivered through the usual sequence of tool calls passes. On the public data this was 3 to 13% of telecom failures and 8 to 30% of retail ones. Those numbers go on the certificate too. If they are too high for your use, put a text check or a second model on top.
It sits in front of it. The grader reads what the engine flags, plus the slice the bound says you still owe. On telecom traffic that cut grading calls by 40 to 56% at a 10% bound.
Probably not, and we have the data to say so. On 3,665 public coding-agent runs the structure of a run in progress did not predict whether it would run away. The method works where a task has an expected shape: customer service, back-office flows, clinical assistants.
Events, not transcripts: actor, step type, tool name, timestamp, token count. If you log OpenTelemetry GenAI spans or use a framework that records tool calls, you already have them. We also need a set of conversations you know went wrong. A few dozen per task type is enough to start.
Yes. The same sessions, in any order, give the same model hash and the same decisions. Every decision is written to a hash-chained log and replays bit for bit after save and load. Someone who does not trust us can rerun the whole thing.