Success Metrics

In plain words

Your mess warden says food improved because 1200 plates were served. True number, wrong question. Same thing happens with agent dashboards full of tokens and latency. So start from the goal, turn it into plain questions, and only then pick numbers. Add a guardrail so a fast agent cannot also be an unsafe one, and write down pass and fail before you see any result.

How it flows

A tiny example

plan = [
    metric("correct_rate", kind="outcome", at_least=0.90),
    metric("prod_writes", kind="guardrail", at_most=0),
]
check_plan_has_guardrail(plan)
values = run_replays(plan)
print(decide(plan, values))

Notice the guardrail is part of the plan, not something added after the results look bad.

What you will learn

• How to go from a goal to a question to a metric, in that order.
• What every metric must carry with it before you trust the number.
• Why one good number is never enough, and what to pair it with.
• How to write down pass, fail and unclear before you see any result.

The problem, simply

Think about your college mess. The warden says food quality has improved. You ask how they know. They say, "We served 1200 plates yesterday, highest ever."

See, plates served is a real number. It is also useless here. Students eat there because there is no other option, not because the sambar got better.

The same thing happens with agents. Your team builds an agent that reads server logs and tells the on-call engineer which service broke. Somebody puts up a dashboard showing tokens used, requests per hour, average response time. All green. Nobody can answer the only question that matters: is the engineer finding the broken service faster, without doing anything dangerous?

That gap is what this lesson fixes. Basically, you decide what you are measuring and what number would make you stop the project, before you have a single result to look at.

The idea

Goal, then question, then metric

Never start with "what can we log". Start with what you actually want.

Say the goal is: reduce the time to identify which service is broken, without letting the agent do anything unsafe.

From that goal, ask plain questions. How fast is the right service found? How often is it actually the right one? Does the agent stay read-only? Are engineers now ignoring alerts because the agent cried wolf?

Only now pick numbers that answer those exact questions. This order has a name that shows up in interviews: Goal-Question-Metric, an old software measurement method. The whole point is that a metric is downstream of a decision, not the other way round.

Every metric needs a contract

A metric is not a word. "Accuracy" is a word. A metric is a small contract with six things filled in:

• Name — median_identification_seconds, not "speed".
• Direction — at most, or at least.
• Threshold — 120 seconds.
• Window — over ten replayed incidents.
• Source — the replay event log.
• Population — on-call engineers in the pilot batch.
• Kind — outcome, guardrail, or counter-metric.

IMPRemember: without a source and a window, nobody can reproduce your number. Without a threshold, the number cannot decide anything.

Three kinds of metrics

An outcome metric asks: did the thing we wanted actually get better? Median identification time.

A guardrail asks: did a hard rule stay true? Zero writes to the production database. This is a constraint, not a target — you do not "improve" it, you just must not break it.

A counter-metric asks: did we push the pain somewhere else? Maybe the engineer is faster, but now a junior spends thirty minutes every morning cleaning up the agent's false alerts. Speed went up, total work went up too.

Warning

Warning: An agent that is fast and wrong looks excellent on a dashboard with only one metric. Guardrails exist precisely because outcome metrics are easy to game.

Offline replay or a real pilot?

Offline replay means running the agent over recorded past incidents. It is cheap, repeatable, and good for covering weird edge cases.

A small pilot means a few real engineers using it for two weeks. That is the only way to see trust, habit and workflow effects — whether people actually open the agent's suggestion or scroll past it.

Neither replaces the other. Use the cheapest evidence that can answer today's decision. Do not put real users in front of the agent just because the code happens to be ready.

Worked example

Suppose Priya is shipping this incident agent for a fintech team.

Her plan, written before any run: outcome — correct service identified in at least 90% of replays, and median time at most 120 seconds over ten replays from the event log. Guardrail — zero production writes. Counter-metric — alert dismissal rate must not go above its current level.

Her decision rules, also written first. Pass: correct rate 0.9 or above and median at or under 120 seconds. Fail: any production write at all, or correct rate below 0.75. Ambiguous: a small improvement with wide spread across the ten replays — then she runs a bigger replay set instead of arguing.

The result comes back: 0.87 correct, 95 seconds median, zero writes. That is the ambiguous zone. Because she wrote the rule earlier, the team runs forty more replays instead of deciding that 0.87 is "basically 0.9".

Build it

This little program takes a measurement plan, checks that the plan itself is valid, then judges the observed values against it.

"""A tiny measurement-plan checker. Standard library only."""

# Each metric is a contract: name, kind, direction, threshold, window, source.
PLAN = [
    {"name": "correct_service_rate", "kind": "outcome",
     "direction": "at_least", "threshold": 0.90,
     "window": "10 replays", "source": "replay_log"},
    {"name": "median_identification_seconds", "kind": "outcome",
     "direction": "at_most", "threshold": 120,
     "window": "10 replays", "source": "replay_log"},
    {"name": "production_writes", "kind": "guardrail",
     "direction": "at_most", "threshold": 0,
     "window": "10 replays", "source": "audit_log"},
    {"name": "alert_dismissal_rate", "kind": "counter",
     "direction": "at_most", "threshold": 0.30,
     "window": "2 weeks", "source": "ops_dashboard"},
]

OBSERVED = {"correct_service_rate": 0.87,
            "median_identification_seconds": 95,
            "production_writes": 0}   # counter-metric not collected yet

REQUIRED = ("name", "kind", "direction", "threshold", "window", "source")

def validate(plan):
    """A plan is usable only if every field is filled and a guardrail exists."""
    problems = []
    for m in plan:
        for field in REQUIRED:
            if m.get(field) is None:
                problems.append(m.get("name", "?") + " is missing " + field)
    if not any(m["kind"] == "guardrail" for m in plan):
        problems.append("no guardrail metric: speed could hide unsafe behaviour")
    return problems

def judge(metric, value):
    """Inclusive comparison, so the threshold itself counts as a pass."""
    if value is None:
        return "missing"
    if metric["direction"] == "at_least":
        return "pass" if value >= metric["threshold"] else "fail"
    return "pass" if value <= metric["threshold"] else "fail"

problems = validate(PLAN)
print("plan problems:", problems or "none")

results = {}
for m in PLAN:
    verdict = judge(m, OBSERVED.get(m["name"]))
    results[m["name"]] = verdict
    print(f"{m['name']:32} {verdict:8} (source={m['source']})")

# Decision rule, written before the numbers were seen.
if results["production_writes"] == "fail":
    print("DECISION: fail (guardrail broken)")
elif "missing" in results.values():
    print("DECISION: ambiguous (evidence incomplete, collect it before deciding)")
elif all(v == "pass" for v in results.values()):
    print("DECISION: pass")
else:
    print("DECISION: ambiguous (widen the replay set, do not move the threshold)")

Run it and look at three things. The plan itself passes validation because every field is filled. The correct-service rate fails at 0.87 while time passes at 95 seconds. And the counter-metric was never collected, so the verdict is ambiguous, not pass — missing evidence is a real result.

Tip

Tip: Delete the guardrail entry from PLAN and run again. The plan becomes invalid even though both outcome metrics are still there. That is the lesson in one command.

Where you will see this

• Coding agents like Claude Code and Cursor are judged on tasks actually completed, with a guardrail that they do not silently break existing tests.
• Customer-support bots are measured on issues resolved without a human, with a counter-metric on how many escalations the human team receives afterwards.
• Swiggy or Zomato style assistants track order completion, with a guardrail on wrong-item orders that cost refunds.
• Any A/B test at a product company: one primary metric, and a set of guardrails that can stop the rollout by themselves.
• College placement dashboards: offers made is the outcome, but nobody tracks the students who dropped out of the process.

Common mistakes

• Choosing metrics from what is easy to log. Tokens and latency are easy, so they end up on the dashboard, but neither one can change your decision about shipping.
• One number only. A single outcome metric with no guardrail rewards a fast agent that does dangerous things.
• Deciding the threshold after seeing the result. The number will always drift towards whatever the build achieved. Write it down first, in a file, with a date.
• Skipping source and window. Six months later nobody can reproduce "94% accuracy" because nobody knows which data it ran on.
• Jumping straight to real users. If a replay over recorded incidents can answer the question, do that first. Real users are the expensive evidence.

If they ask in an interview

Q: How would you measure whether your agent is working?

A: I would start from the goal, turn it into two or three plain questions, and only then pick metrics — that is the Goal-Question-Metric order. Each metric gets a direction, a threshold, a window and a source, so it is reproducible. And I would pair the outcome metric with at least one guardrail.

Q: What is a counter-metric and why do you need one?

A: A counter-metric catches cost that gets pushed elsewhere. For example, an agent halves the on-call engineer's diagnosis time, but a junior now spends an hour a day fixing its false alerts. Without the counter-metric, the team reports a win while total effort actually went up.

Q: Offline evaluation or a live pilot — which one do you pick?

A: Whichever is the cheapest evidence that can answer the current decision. Replay over recorded cases is repeatable and good for edge coverage. A small pilot is the only way to see trust and workflow effects. If the question is about human behaviour, replay cannot answer it.

Try these

1. Take one project you built and write its goal in one sentence. Derive three questions from it, then one metric per question.
2. Add a counter-metric to the code above that catches work shifted to a different person, and give it a source and a window.
3. Fill in source, population and window for every metric in your plan. Any metric where you cannot fill these in — delete it.
4. Write your pass, fail and ambiguous rules in a text file, then generate some random values and see how often you are tempted to argue with your own rule.

Words, simply

Quick recap

• Goal first, then questions, then metrics. Never start from what is easy to log.
• Every metric carries a direction, threshold, window and source, or it cannot be reproduced.
• One outcome metric plus a guardrail plus a counter-metric, with pass and fail written before you look.