:
November 5, 2026
.png)
Picture the founder of Acme Outfitters, a made-up online clothing store, asking the AI agent in their accounting software to fix something in the books: payments to one vendor keep landing in software expense, but they belong under contractors. Could the agent change that?
The agent does what a careful bookkeeper would. It reads the existing rules, drafts a new one, tests it on past payments, shows the founder the result and publishes it once they approve. It replies: "Done. The rule is published, so these payments will go under contractors from now on." That's one turn, in the everyday sense of taking turns in a conversation: the founder asks, and the agent does its work and replies.
The founder's next message: "No, it still shows under software."
By every measure a typical engineering dashboard tracks, nothing went wrong. The agent replied without crashing. It didn't keep the founder waiting. No errors were logged. And yet the founder did not get what they asked for, and the only record of that is a sentence in a chat.
Neither of them did anything wrong. The agent answered the question it heard, and the founder meant the payments already in the books as well.
Now imagine the founder never wrote back. They close the month that afternoon and send their board a profit and loss statement with the contractor spend still sitting under software. Nothing errors, and every dashboard stays green. Nobody finds out until someone checks the numbers by hand, if anyone does.
That silent version is the one we worry about most: an AI that says "Done" but hasn't actually completed the task, and nothing in the logs flags it. The founder's "No" is the lucky version, because it leaves evidence behind. Read that evidence well and you can fix causes that probably also hit the people who never write back.
Search engines faced this problem decades ago. A results page can load in a fraction of a second and still fail to answer the question that the person searched for, but the server's record of that search looks the same either way. What gives it away is what the person does next: they come straight back from the result they clicked, or they type the same question again in different words. Research on search logs has found that a reworded query within a few minutes is a strong sign of an unhappy searcher, and that models built on signals like it beat models built on clicks alone. The user of an agent leaves the same kind of trail, except that it's written in plain language in their next message.
This is the core difficulty with observing agents. When ordinary software breaks, it usually says so: an error, a crash, a timeout. An agent improvises as it goes: it reads some context, calls a tool, reconsiders, calls another. Nearly every turn finishes. The question that matters, whether the finished turn helped, never shows up as an error. Often the answer isn't in the agent's own output either, but in what the user does next.
When we launched our agent at T:0, our first dashboard was a tab on our internal admin console, next to system monitoring, with the numbers you would expect: turns per day, active users, cost per turn, response time, and the share of turns that finished, errored or were stopped. All of them describe how the system ran, not whether it helped, and none of them would have caught a conversation like this one. This post is about how we built our agent analytics to catch that kind of failure: one that finishes cleanly and shows up only in what the user says next.
If you're not an engineer, the short version: we read what people say after the agent answers, we judge it with fixed rules rather than another AI, and we're explicit about what we can't see. The engineering detail follows, for anyone building agents of their own. If you only want to see how we spot those failures, skip ahead to "Reading failure from what happens next".
Before deciding what to capture, we pictured an engineer on a Monday morning who shipped a prompt change on Friday. She wants to know whether the agent is falling short more or less often than before, and for how many people, because one struggling account and a problem that hits everyone need different responses. She wants to know which job is failing, since categorizing transactions, writing accounting rules, building reports and answering questions about the ledger each break in different ways and get fixed in different places. She wants to know what people did after a bad answer: recovered, kept trying or left. And she wants to know where the time and money went.
For each of those answers, she needs to open the conversations behind the number, because a count nobody can inspect is hard to act on.
Before we could judge a reply by the message that followed it, we had to be able to find that reply again, along with everything the agent did to produce it. Our agent works inside a company's accounting records, where it can look things up and draft changes, though nothing changes until the user approves. For every turn, we keep a step-by-step recording of what the AI was asked, what it looked up or drafted, and what came back, plus a few labels of our own, such as what kind of job it was and how it ended.
Every reply links straight to its recording, even when the agent pauses for the user's approval and picks up again later. If recording ever fails, the agent keeps answering and we lose only that turn's recording.
The first version of our new analytics read its numbers from these recordings. It lasted about a day. Recordings only existed from the day we started making them, so everything before that was invisible, and the engineer from Monday morning couldn't tell whether Friday's change had made things better or worse without a "before". But the same facts (which tools a turn used, whether it hit an error, whether the user approved) had been sitting in our own conversation records all along. So we moved every number there and calculated it with the same code across every turn we had ever stored. The recordings kept one job: showing exactly what happened in a single turn when someone needs to look.
Run the founder's reply, "No, it still shows under software", through rules like ours and it triggers two of them. It opens with "no" within half an hour of the agent's answer, which counts as the user correcting the agent. And it says nothing has changed right after the agent published a rule, which counts as an action that didn't work.
Other rules watch for the agent saying it can't do what was asked or apologizing and correcting itself, and for the user asking the same thing again in nearly the same words. The agent's own steps count too: a lookup that fails or comes back empty, a change the user rejects, or the same step repeated three times. Any finished turn that triggers one of these is flagged: something suggests it didn't help.
We chose simple rules over an AI judge on purpose. An AI judge would catch subtler cases, but it doesn't always give the same answer twice, and rerunning it over every past conversation is slow and costly. If the failure rate rose, we couldn't tell whether the agent had changed or the judge had. Rules grade last spring's conversations exactly as they grade today's, and when we tighten one, it regrades every past conversation too, so the trend never jumps just because a rule changed.
We leave out turns we can't fairly grade: an outage at the AI provider we rely on, a user pressing stop, or a turn still waiting for approval. They aren't hidden: the page shows how many outages and stops were left out, and a conversation that ends on one is still counted as unresolved. A quick follow-up that needs no new lookup, such as "Thanks, can you explain that?", counts toward the job before it, so a confusing answer to it counts against that job.
Each flagged turn also gets a likely cause, such as bad source data, a misread request or a broken tool, based on which rules were triggered. That tells us whether the fix belongs in the data, the agent's instructions or the tool.
Grading single turns tells you where the agent stumbled. It doesn't tell you whether the person got what they came for. For that, we judge the whole conversation by how it ended. If something went wrong along the way but the last turn was clean, it's a recovery candidate. If it ended on a flagged turn and the user hasn't come back for a day, it's unresolved. If nothing was flagged, it's no issue detected. Conversations that are still open get one of two holding labels: waiting for the user's approval, or too soon to tell.
In the Acme case, the agent checks, sees that the founder also meant the payments already in the books, and, once the founder approves, moves those to contractors too. If that reply triggers no rule, the conversation is a recovery candidate.
The names are careful. There's no "resolved". Remember the founder who never wrote back: to every rule, that conversation looks like success, so the best we can honestly say is that no issue was detected. "Recovery candidate" is hedged for the same reason. A clean last turn suggests the user got what they needed, but it would look the same if they had simply given up.
A failure rate built from rules is easy to over-read. Rules miss real failures, and they also flag good answers. So the headline number is labeled for what it is: turns flagged by our rules. An earlier version called it the "minimum failure rate", but false alarms mean the true rate could be lower, so we dropped the name.
People check the rules. Say a reviewer opens a flagged turn and finds the user wrote "No, that's right, thanks." The "no" triggered the correction rule, but nothing was wrong, so she marks it a false alarm and the turn leaves the count. The rule's original call is kept, so once thirty flags have been reviewed, the page shows how often the reviewed flags held up, and the false alarms point to the rule that needs tightening. Because reviews remove false alarms, a busy week of reviewing can also make the line dip when the agent hasn't changed, so the page shows how many flags were reviewed. To catch the opposite mistake, reviewers also see a few recent conversations the rules passed as clean, and any failure they find there joins the count.
Every rate carries a margin of error, as a poll does, and anything based on fewer than twenty turns is marked too small to trust and kept off the trend charts. Problems to fix are ranked by how many conversations they left unresolved and how many customers they touched, and a separate panel shows where the time and money went.
Back to the engineer from Monday morning. If a number looks off after Friday's change, she clicks it and sees the conversations behind it: what each user wanted, how it ended and which rules were triggered. She opens one as a turn-by-turn timeline, and one more click shows exactly what the AI saw and did.
There's one place we do use an AI model: writing short notes each week on where the agent is falling short. The risk is lower there because the notes feed no number. If the model changes, the wording of the notes changes, but no trend line moves.
We still limit what it sees and what it can claim. It reads the users' own words, how each turn ended and which job it was doing, but not the agent's replies or any company's accounting records. Without the agent's side, it can't see why things broke, so it's told to describe what users ran into, hedge its claims and cite only conversations it was given. AI can make things up, so any cited conversation we can't find is dropped, and findings are ranked by how many conversations they cite.
In the Acme example, nobody files a complaint. The founder says the change hasn't worked, the agent fixes it, and they move on. In a dashboard that only counts errors, that conversation looks like two successful turns.
In analytics like ours, it shows up as a recovery candidate next to any others like it, where an engineer can see that the agent covered future payments when the founder meant past ones too. A pattern like that has a clear fix: have the agent ask which payments the founder means before it says "Done". That fix also helps the founders who never wrote back.
Search engines learned to treat the next query as feedback. For an agent, that feedback is the user's next message, and our job is to read it.
.png)
.avif)