Talking in percentages

TL;DR. AI bugs are probabilistic, so one observation doesn't tell us how often a behaviour actually happens. Anyone investigating a pattern should be able to quickly turn it into a measured rate, so we can quantify issues, spot trends, prioritise, and tell whether a fix worked - for every shape of behaviour.

The problem

Traditional bug reporting works because traditional bugs are deterministic. Someone observes the issue, an engineer reproduces it, the root cause gets fixed, and within minutes we can tell whether the fix worked.

This loop falls apart for AI behaviour. Agents fail probabilistically: failures happen on some tickets and not others, often in slightly different ways. When someone reports that the agent hallucinated a refund policy, we are looking at one observation of a behaviour that might happen 1% of the time, or 50%, and from the report alone there's no way to tell which. The same anecdote can be a one-off or a daily occurrence, and the surface evidence looks identical.

Almost every downstream activity depends on a rate we don't have. Engagement: when leadership or a customer asks "how big is this?", a screenshot isn't an answer. Validation: when we ship a fix, there's no before and after, and we judge by whether the demos look better. Regression detection: when the behaviour creeps back in, nobody notices until another ticket gets escalated through the same anecdotal pipeline.

Two examples

Measuring a rollout. During a rollout we needed to know whether a change was reducing the rate at which the agent claims actions it never performed. We already had a signal for it, but on first inspection it ran on too few tickets in the rollout cohort to show anything meaningful. When we tried to tune it, the check turned out to be missing real cases the team had already found by hand. Once we fixed that, the opposite problem appeared: the check started firing on cases that weren't actually failures. Weeks of engineering time went into making one signal trustworthy enough to use as the project's headline metric, and running it at the sampling rate we needed created capacity problems of its own.

Product feedback. Feedback on an AI product has the same problem in a different shape. It arrives through Slack, docs and DMs from the people deploying it and the people using it, and some patterns are obvious: this feature is too inaccurate, that button is too eager. We can describe the shape but not the prevalence. "Too eager" might be one user feeling strongly about it, or something every customer will hit on day one, and we can't tell from the channel. Prioritisation defaults to whoever is loudest, fixes ship without a baseline, and after a change we can't tell whether the situation actually improved.

Why anecdotes struggle

1. The behaviour is unanticipated. The agent interpreted the situation in a new way we couldn't have written a check for ahead of time, because we didn't know to look for it. Existing checks are tuned for things we already knew about.
2. One observation is not a rate. Every instance is a fire if we can't see how common it is, especially when a customer raises it.
3. Qualitative judgement is itself noisy. Even when we sample tickets and have a human grade them, the signal is small, judgement varies between graders and over time, and a true 2% rate is indistinguishable from noise across any sample we can reasonably collect by hand.

When we do decide a behaviour matters enough to measure, building a trustworthy instrument for it is involved enough to be a project on its own. The structural problem isn't whether we can measure agent behaviour; it's that measuring any new one is too expensive to do for the long tail.

What it should take

Anyone investigating a behaviour should be able to quickly turn it into a measured rate, for every shape of behaviour. We already see how well this works when rates are cheap to compute. "This link fails on 4% of messages" is a clear priority, with a baseline any fix has to beat. AI behaviour doesn't give us that rate for free, because the failures aren't observable in logs the way a broken link is. Instead of stopping at "the agent hallucinated a refund policy", we should be able to ask "how often does this happen?" inside the same investigation and get an answer like "3% of escalated tickets" or "8% of refund tickets where the customer asks twice". A rate lets us prioritise against everything else competing for attention, set a baseline that fixes have to beat, and detect regression once a fix has shipped.

Making this cheap and credible enough to use routinely takes more than a judge prompt. The examples above point to three requirements any solution has to meet.

1. Self-serve authoring with a tuning loop. Anyone investigating a behaviour - engineer, PM, the person deploying the agent - should be able to describe it, optionally paste a few real tickets as seed examples, get a draft judge prompt back, run it against a sample, walk through the false positives and false negatives to refine it, then re-run. Today this tends to be a side project: someone builds the recall and precision loop by hand from a labelled set. The whole authoring motion should be quick enough to happen inside an investigation.
2. Trend visibility. A new signal is only useful for monitoring once we can compare today against the last 30 to 90 days, so backfilling history has to be part of creating the signal rather than a follow-up ticket for an engineer.
3. High coverage at a controlled cost. A low steady-state sampling rate works for ongoing monitoring but isn't enough during a rollout. Sample 5% of tickets and look at a change rolled out to 5% of traffic, and you're left with about 0.25% of tickets - too few to draw conclusions from. Signals tied to active investigations need to ramp up to 100% for the duration of the work and back down afterwards, inside a cost ceiling that keeps them from starving every other check.

The capability has to be cheap enough - in time and in cost - that we use it routinely. If creating a signal takes more than an afternoon, we end up back at anecdotes; if running it at the sampling rate we need blows the budget, we end up running it too thin to be useful. The dataset, the judge, the tuning loop and the dashboard should fall out of the investigation.