Observability Platforms

In plain words

Your agent is like a hostel mess register. It records what was served, but nobody scores whether the food was good. An observability platform fixes that: it takes in every step of every agent run, attaches a quality score, and keeps the prompt version next to it. So when users complain on Thursday, you can point at the exact change that broke it on Tuesday.

How it flows

A tiny example

for span in incoming_spans:
    if guardrail_trips(span):
        redact(span)
    score, reason = judge(span)   # LLM-as-a-judge on a rubric
    store(span, score, reason, span["prompt_version"])

for version, scores in group_by_version(store).items():
    print(version, average(scores))

Notice the score and the prompt version are stored together — that single pairing is what lets you find which prompt change caused a regression.

What you will learn

• What an agent observability platform actually does for you.
• The three popular open-source ones — Langfuse, Phoenix, Opik — and what each is best at.
• Why saving traces without scoring them is a waste of money.
• How to build a tiny trace collector plus a scoring step in plain Python.

The problem, simply

See, think about the hostel mess. Every day 300 students eat there. The manager keeps a register of how much rice, dal and sabzi went out. Nice register. Very neat.

But nobody ever asks the students, "Was today's food good?" So the register grows fat and the food quietly becomes worse. The manager can tell you what was served on 12 August. He cannot tell you when the taste started dropping.

Your agent in production is exactly this mess. In the previous lesson of this module you learned how to record what the agent did — every model call, every tool call, every step. That register is called a trace. Recording is the easy half.

The hard half is the Monday morning question: "Users complained yesterday. Which change broke it?" To answer that, somebody must score the runs, keep prompt versions next to the traces, and show you where quality fell. That somebody is an observability platform.

The idea

Trace, then judge

A trace is the full story of one agent run, broken into small pieces called spans — one span for the model call, one for the search tool, one for the database query. A platform takes these spans in, groups them by session, and stores them.

Then comes the part people skip. You attach a score to each run. The score can come from a user thumbs-down, from a rule you wrote, or from LLM-as-a-judge — a second model whose only job is to read the agent's answer and mark it against a rubric, like a paper checker.

IMP

Important: tracing without evaluation is just expensive logging. You can see every run and still not know which runs were bad.

A worked example

Suppose Priya builds a support agent for a Flipkart-style store. On Tuesday she changes the system prompt to make replies shorter. Sales team complains on Thursday that the bot is "rude now".

With only a register, Priya reads 200 chat logs by hand. With a platform, she opens the dashboard, sees the average tone score fall from 4.4 to 3.1, and sees the drop start exactly where prompt v7 replaced v6. Two minutes, not two days. That is the whole value: prompt versions tied to traces, so you can bisect.

The three platforms

Langfuse — the all-rounder. Tracing, versioned prompts with a playground, evaluations, and session replay (stepping through an old run like a video). It is MIT licensed, which is the most permissive of the three. Since June 2025 even its earlier paid pieces — the LLM-judge, annotation queues, prompt experiments, the playground — are open under MIT. Pick this if you want one tool for everything and you care about prompt management.

Arize Phoenix — the drift detective. It is stronger on agent-specific evaluation: clustering similar traces together, spotting anomalies, and checking RAG relevancy (in a retrieval system, did the fetched documents actually match the question?). It auto-instruments your code through a convention called OpenInference, so you write less wiring code. It has no prompt versioning, so people run it beside another tool. Its license is Elastic License 2.0, which is not fully open — worth checking with your company before you commit.

Comet Opik — the optimizer. It runs A/B experiments on prompts automatically, enforces guardrails (redacting personal details like phone numbers, blocking off-topic replies) at logging time, and uses an LLM-judge to catch hallucinations. Apache 2.0 licensed.

All three speak the same span format, so all three can push data into Datadog or New Relic if your ops team already lives there. Most teams run one of these now — around 89% of organisations report having agent observability in place, and quality problems are their top production headache.

Warning

Warning: Comet's own published benchmark shows Opik logging and evaluating in about 23 seconds against Langfuse's 327 seconds. That is a vendor measuring itself. Treat such numbers as a hint, and measure on your own data before deciding.

Build it

This is a tiny version of what these platforms do. Fake spans go in, a scripted "judge" scores them, and a dashboard summary comes out.

# A toy observability platform: ingest spans, score runs, print a dashboard.
from collections import defaultdict

# Pretend spans coming from an agent. In real life these arrive over the network.
SPANS = [
    {"session": "s1", "prompt_version": "v6", "answer": "Your refund of Rs 499 is processed.", "user_msg": "refund status"},
    {"session": "s2", "prompt_version": "v6", "answer": "Order 8812 ships tomorrow by 6 pm.", "user_msg": "when will it ship"},
    {"session": "s3", "prompt_version": "v7", "answer": "No.", "user_msg": "can I cancel"},
    {"session": "s4", "prompt_version": "v7", "answer": "Call 9876543210 for details.", "user_msg": "cancel order"},
    {"session": "s5", "prompt_version": "v7", "answer": "Cannot say.", "user_msg": "refund status"},
]

def judge(span):
    """A fake LLM-as-a-judge. Returns a score out of 5 plus a reason."""
    ans = span["answer"]
    if len(ans) < 12:                       # too short to be helpful
        return 2, "too_short"
    if any(ch.isdigit() for ch in ans) and "Rs" not in ans and "Order" not in ans:
        return 2, "possible_pii"            # a bare number looks like a phone number
    return 5, "ok"

def guardrail(span):
    """Blocks answers that leak a 10-digit number. Runs before storing."""
    digits = "".join(c for c in span["answer"] if c.isdigit())
    return len(digits) >= 10

# Ingest: score every span and bucket it by prompt version.
by_version = defaultdict(list)
reasons = defaultdict(int)
blocked = 0
for span in SPANS:
    if guardrail(span):
        blocked += 1
    score, reason = judge(span)
    by_version[span["prompt_version"]].append(score)
    if reason != "ok":
        reasons[reason] += 1

# Dashboard: average score per prompt version, failures, guardrail trips.
print("=== Toy agent dashboard ===")
for version in sorted(by_version):
    scores = by_version[version]
    avg = sum(scores) / len(scores)
    print(f"prompt {version}: runs={len(scores)}  avg_score={avg:.2f}")

total = len(SPANS)
bad = sum(1 for s in SPANS if judge(s)[0] < 3)
print(f"failure rate: {bad}/{total} = {100 * bad / total:.0f}%")
print(f"guardrail trips: {blocked}")
print("top failure reasons:", dict(reasons))

Look at the two average scores. Version v6 sits near the top and v7 falls hard — that is the regression, and you found it without reading a single chat by hand. The guardrail count tells you one answer leaked a phone number. The reason counts tell you why runs failed, not just that they failed.

Where you will see this

• Coding agents like Claude Code and Cursor, where teams trace long multi-step sessions to find where the agent went off track.
• Customer support bots at banks and e-commerce companies, where every reply is scored for tone and correctness before anyone trusts it.
• Swiggy or Zomato style assistants, where a wrong order-status answer must be caught the same day, not next month.
• Internal RAG search over company documents, where the main question is "did we fetch the right document" rather than "did the model write nicely".
• Any team already on Datadog or New Relic that wants agent data in the same dashboards as their servers.

Common mistakes

• Collecting traces with no evaluation plan. You pay storage and get a pretty timeline, but nothing tells you quality dropped. Decide the rubric before you start collecting.
• Writing your own LLM-judge with no grounding. A judge that only reads the answer and nothing else slowly starts approving everything. For factual claims the judge needs a real tool — a search, a database lookup — to check against.
• Not storing the prompt version with the trace. Then when production breaks you cannot bisect. You will be guessing which of last week's five prompt edits did it.
• Trusting vendor benchmarks. Every company's numbers make it look fastest. Run all three on a few hundred of your own traces.
• Ignoring the license. Elastic License 2.0 has restrictions that a plain MIT or Apache 2.0 license does not. In a company review this can block your whole plan late.

If they ask in an interview

Q: What is the difference between logging and observability for an AI agent?

A: Logging records what happened. Observability lets you ask new questions of that record — which sessions failed, why, and what changed. For agents that means traces plus scores plus prompt versions, so you can go from a complaint to a root cause.

Q: How would you catch a quality regression after a prompt change?

A: Tag every trace with its prompt version and attach an evaluation score to each run, either from user feedback or an LLM-judge on a fixed rubric. Then compare average score per version. The version where the score drops is your culprit, and you can replay those sessions.

Q: What is LLM-as-a-judge and where does it fail?

A: It is a second model that scores the agent's output against a rubric, like an examiner. It works well for tone, format and scope. It fails on facts, because without an external tool to verify against, the judge tends to approve confidently wrong answers.

Try these

• Take the toy code above and add a third prompt version with your own five fake answers. Check that the dashboard correctly shows which version is worst.
• Write a rubric for a domain you know — say a college admission helpdesk bot. Three criteria, each scored 1 to 5, with one line describing what a 5 looks like.
• Add a second guardrail to the code that blocks answers containing an email-like string, and print how many runs it caught.
• Change the judge so it also flags answers that do not mention anything from the user's question, then see how the failure reasons shift.

Words, simply

Quick recap

• A trace tells you what happened; a score tells you whether it was good. You need both.
• Langfuse for all-in-one with prompt management, Phoenix for RAG and drift, Opik for optimization and guardrails.
• IMPRemember: always store the prompt version with the trace, otherwise you can never bisect a regression.