Reviewer Agent

In plain words

You proofread your own assignment at 3 a.m. and it looks perfect, then your guide finds three mistakes on page two. Agents are the same. So you add a second agent whose only job is to read the builder's work and grade it against five fixed questions. It can read everything, but it cannot change a single line.

How it flows

A tiny example

artifacts = collect(diff, tests, notes)
scores = {}
for question in RUBRIC:
    scores[question] = judge(question, artifacts)  # 0, 1 or 2
total = sum(scores.values())
verdict = "pass" if total >= 7 else "fail"
write_report(scores, total, verdict)  # reviewer writes, never patches

Notice the reviewer only ever writes a report; it never touches the diff it just read.

What you will learn

• Why the agent that writes the code cannot be trusted to grade it
• How to write a small review rubric with fixed questions instead of vibes
• How a reviewer agent reads the builder's work and writes a verdict
• Where a reviewer sits next to the verification gate, not on top of it

The problem, simply

Think of your final year project. You wrote the code, you wrote the report, and then you proofread your own report at 3 a.m. It looked perfect. Next morning your guide opened page two and found the same mistake in three places.

That is not a talent problem. Your eyes fill in what you meant instead of what is actually there.

Agents have the same weakness. You ask an agent to fix a bug. It edits four files, runs the tests, and says "done". A verification gate checks the boring facts: tests ran, tests passed, no files touched outside the contract. All green, so you merge.

Two days later you find it fixed the wrong half of the bug. The tests passed because the agent wrote tests for the half it fixed. Passing is necessary, not enough.

IMP

Note: The verification gate answers "did the work follow the rules?". Nobody in the loop was answering "was this the right work at all?".

The idea

A second agent with a different job

The fix is a reviewer agent. Same as asking Rahul from the next room to read your report before you submit. Fresh eyes, no attachment to the code.

The reviewer gets a different system prompt, different inputs, and one rule that matters most: it can read the diff but it cannot edit the diff.

If the reviewer is allowed to fix things itself, it stops being a reviewer. It becomes a second builder, and now nobody is checking. When the report says "this is wrong", the builder takes another turn and the reviewer goes back to reviewing.

IMPRemember: the reviewer needs a different role, not a different model. The same model can play both parts. What changes is the prompt, the inputs, and the fact that it holds no pen.

The five-question rubric

"Review this properly" gives you a nice paragraph and no signal. So you fix the questions in advance. Five dimensions, each scored 0, 1 or 2. Total out of 10.

Below 7 out of 10 is a soft fail: send the findings back to the builder. Below 5, or any single dimension at 0, is a hard fail: stop and call a human.

A worked example

Suppose Priya asks the agent to fix a bug where UPI refunds above ₹5,000 are not showing in the user's wallet.

The agent edits the wallet display, adds a test for a ₹6,000 refund, and it passes. Gate is green.

Now the reviewer reads the same artifacts. Problem fit gets 1: the real bug was in the refund webhook, the display was only a symptom. Verification quality gets 1: the test proves the number renders, not that the refund was recorded. Scope, assumptions and handoff get 2 each. Total 8 out of 10, and the report names the two weak spots.

Priya now has one paragraph to read instead of a 400-line diff. That is the whole value.

How teams run this at scale

At small scale, one reviewer with five questions is enough. Large teams split it into specialists: up to seven narrow reviewers running in parallel for security, performance, code quality, docs, release and compliance. Cheap models run the specialists. One expensive coordinator merges their findings, drops duplicates and decides what is serious.

The reviewer has its own bugs

When a model grades text, it is called LLM-as-a-judge, which just means "the model is the examiner". Four biases show up again and again:

• Position bias — the same two answers get different winners if you swap the order.
• Verbosity bias — longer answers get higher scores just for being longer.
• Self-preference — a judge quietly likes output from its own model family.
• Authority bias — a famous name in the text pushes the score up.

Fixes are simple. Score both orderings and only count the ones that agree. Use a short scale that rewards being concise. Rotate which model family judges. Strip names before scoring.

IMP

Important: Keep a calibration set — 10 to 20 old tasks where you already know the right verdict. Every time you change the reviewer prompt, run it over that set. If it agrees with the known verdicts less than 80 percent of the time, fix the rubric before shipping it.

Build it

"""A tiny reviewer agent. Reads builder artifacts, scores 5 questions, writes a verdict."""

DIMENSIONS = ["problem_fit", "scope", "assumptions", "verification", "handoff"]

def score_run(run):
    """Each check is a stub here. In real life these would ask a model."""
    scores = {}
    # Did the change touch the file the task actually pointed at?
    scores["problem_fit"] = 2 if run["task_file"] in run["files_changed"] else 1
    # Did it stay inside the agreed file list?
    extra = [f for f in run["files_changed"] if f not in run["allowed_files"]]
    scores["scope"] = 2 if not extra else 0
    # Were assumptions written down?
    scores["assumptions"] = 2 if run["assumptions"] else 1
    # Does the test mention the thing the task was about?
    scores["verification"] = 2 if run["keyword"] in run["test_name"] else 1
    # Can the next session continue?
    scores["handoff"] = 2 if run["notes"] else 0
    return scores

def verdict(total, scores):
    if total < 5 or 0 in scores.values():
        return "hard_fail"
    return "soft_fail" if total < 7 else "pass"

def review(run):
    scores = score_run(run)
    total = sum(scores.values())
    findings = [d for d in DIMENSIONS if scores[d] < 2]
    return {"scores": scores, "total": total,
            "verdict": verdict(total, scores), "findings": findings}

clean = {"task_file": "wallet/refund.py", "files_changed": ["wallet/refund.py"],
         "allowed_files": ["wallet/refund.py"], "assumptions": ["refunds are async"],
         "keyword": "refund", "test_name": "test_refund_recorded", "notes": "done"}

wrong = {"task_file": "wallet/refund.py", "files_changed": ["wallet/display.py"],
         "allowed_files": ["wallet/refund.py", "wallet/display.py"], "assumptions": [],
         "keyword": "refund", "test_name": "test_amount_renders", "notes": ""}

for name, run in [("clean change", clean), ("right tests, wrong problem", wrong)]:
    r = review(run)
    print(f"\n--- {name} ---")
    for dim in DIMENSIONS:
        print(f"  {dim:14s} {r['scores'][dim]}/2")
    print(f"  TOTAL {r['total']}/10  -> {r['verdict']}")
    print(f"  look at: {', '.join(r['findings']) or 'nothing'}")

Run it and compare the two blocks. The first one scores 10 and passes. The second one edits the display file instead of the refund file, writes no notes, and lands in a hard fail even though its test passed happily. Notice that the report names the exact dimensions to look at, so a human reads four words instead of the whole diff.

Where you will see this

• Claude Code subagents: a reviewer subagent runs after the builder closes a task and posts the rubric scores as a pull request comment.
• Agent frameworks with handoffs, where a builder agent hands the finished work to a reviewer agent, which can hand back with findings or escalate to a human.
• Two-model pairing in real teams: a fast cheap model builds, a stronger model with a small focused context reviews.
• Large engineering orgs running specialist reviewers in parallel with a coordinator on top, so humans only see the deduplicated findings.

Common mistakes

• Letting the reviewer edit the diff. The moment it can patch code, it starts approving its own patches, and you are back to proofreading your own report.
• Asking for a free-form opinion. Without fixed questions you get a polite paragraph that always says "looks good overall". Fixed dimensions force a specific answer.
• Making the reviewer redo the gate's job. The gate already proves the tests ran and scope held. Spending reviewer tokens on that wastes the one thing the reviewer is good at, which is judgment.
• Never calibrating. Prompt changes silently drift the reviewer's standards. Without an old set of known verdicts you will not notice until something bad ships.
• Ignoring that builder and judge share a model family. Self-preference is real; the judge goes easy on its own family's style.

If they ask in an interview

Q: Why can an agent not review its own output?

A: Because it grades against the same understanding it used to build. If it misread the task, it will misread it again while checking. A reviewer with a different prompt, different inputs and read-only access asks questions the builder never thought to ask.

Q: Do you need a different model for the reviewer?

A: No. The same model works as long as the role changes: a different system prompt, only the builder's artifacts as input, and no write access. Many teams do use a stronger model for review, but that is a cost and quality choice, not the thing that makes it work.

Q: How do you know your LLM judge is trustworthy?

A: You keep a calibration set of 10 to 20 past tasks with known correct verdicts and rerun the judge on every prompt change. If agreement drops below about 80 percent, you fix the rubric before shipping. You also test for position and verbosity bias by swapping orderings and checking length effects.

Try these

1. Add a sixth dimension that matters for your own project, then argue in two lines why the existing five do not already cover it.
2. Run the reviewer with a terse prompt and a wordy prompt. Which report would you actually read at 11 p.m.?
3. Add a confidence number per dimension and refuse to emit the verdict when the lowest confidence is under 0.6.
4. Take 10 old commits from any project of yours, write down what you think the correct verdict is, then run your reviewer over them and count the disagreements.

Words, simply

Quick recap

• The builder cannot grade itself; reliability lives in the gap between building and reviewing.
• Five fixed questions with 0 to 2 scores beat any free-form "looks good to me".
• The gate proves the rules were followed; the reviewer judges whether it was the right work at all.