What to instrument on an AI feature

A normal service tells you when it breaks. It throws a 500, the error rate climbs, someone gets paged, and the graph they open is the same graph they have been reading for years.

An AI feature does not do that. It returns a 200 with a confident, wrong answer. Latency is fine. The instance is healthy. The first signal is a support ticket six weeks later saying the summaries have gotten worse — and nothing in your telemetry can confirm or deny it.

The fix is not a better model. It is deciding, before launch, what the feature is going to tell you about itself.

Four signals we wire before anything ships

• Cost per request, attributed to the feature rather than to the account. A single unbounded retry loop can double a monthly bill quietly, and an account-level total will not show you which feature did it.
• Latency at p95 and p99, measured where the user waits — including retries, tool calls, and whatever the code does after the model returns. The provider’s number is not the user’s number.
• Refusal and fallback rate: how often the model declined, timed out, or the code quietly took the safe path instead. This one is usually not measured at all, and it is the first to move when something upstream changes.
• Output-shape failures: the response that did not parse, missed a required field, or returned a value outside the allowed set. It is the one quality-adjacent thing a machine can judge without a human, so judge it.

None of the four measures whether the answers are any good. That is deliberate. These are the signals a computer can decide on its own, they move before the complaints arrive, and they cost an afternoon to add.

Keep the input, not just the answer

Log the model identifier, the prompt version, and the input alongside the output. Sample heavily rather than keeping everything — a small percentage of ordinary traffic plus every request that failed a shape check will answer most questions you will actually ask. Decide the retention window and the handling of anything sensitive at the same time, not later.

The reason is regression. When output gets worse, the useful question is which of the three moving parts moved: the prompt, the model version, or the traffic. Without the input stored next to the version that produced it, all three look identical from the outside. This is the operational half of treating prompts as versioned artifacts — the version number only helps if it is attached to the output it produced.

Alert on the boring ones. Review the rest.

Cost, latency, refusals, and parse failures all have thresholds, which means they can page someone. Set the thresholds on day one, when you still remember what normal was supposed to look like. A threshold picked after the first incident is picked to explain the incident.

Quality cannot page anyone, so it gets a cadence instead: a standing review of sampled outputs against the accuracy bar the feature was funded on. Thirty minutes a week, one person, real examples. Teams that skip this are not making a measurement tradeoff — they have decided to find out from a customer.

A dashboard nobody opens is not observability

The most common version of this failure is not missing instrumentation. It is four good panels on a board with no name attached and no day it gets read. Write down who owns the feature after launch and when they look, and put that in the estimate with everything else.

We wire this in as part of our cloud, DevOps and platform work, on our own products and on client projects, because the alternative is finding out about a degradation from the person paying for it.