Verification Gates

In plain words

In a lab exam the evaluator runs your program before writing Pass in the register; your own confidence counts for nothing. A verification gate is that evaluator for an agent. It is a small deterministic program that reads the scope report, the feedback log, the rule report and the diff, and answers one question: is this task really done? If any block-level check fails, the answer is no.

How it flows

A tiny example

def verify(task_id, art):
    findings = []
    findings += check_acceptance(art["feedback"])
    findings += check_scope(art["scope"])
    findings += check_rules(art["rules"])
    blocked = [f for f in findings if f[0] == "block"]
    return {"task_id": task_id,
            "passed": not blocked,
            "findings": findings}

Notice there is no model anywhere inside verify: the same artifacts always produce the same verdict.

What you will learn

• Why an agent must never be the one to say "my work is done".
• What a verification gate is, and the artifacts it reads.
• The difference between a warn finding and a block finding.
• How to write a small gate in plain Python and get one clean verdict.

The problem, simply

See, think of a college assignment submission. You finish the code at 2 am and you tell your friend, "Done, submitted, working perfectly." Your friend believes you. Nobody checks.

Now think of the lab exam. There is an evaluator. He runs your program, checks the output, checks you did not copy from the next system, and only then writes "Pass" in the register. Your confidence does not matter. The register matters.

An agent is that 2 am student. It reads its own diff and says "Looks good." It says "Tests passed" without any record that a test ever ran. It reads the acceptance criteria loosely enough that almost anything counts as finished.

So we do not argue with the agent. We build the evaluator. That evaluator is called a verification gate — a small program that reads what the agent produced and answers exactly one question: is this task actually complete?

The idea

The gate is a function, not a conversation

A verification gate is a plain function. You give it the task's artifacts, it gives you a verdict. No chat, no persuasion, no second opinion.

Which artifacts? In a workbench you already have them lying around from earlier lessons:

• The scope report — which files the agent was allowed to touch, and which it actually touched. (See the Scope Contracts lesson in Module 7.)
• The feedback log — every acceptance command that was run and the exit code it returned. (See the Feedback lesson in Module 7.)
• The rule report — the project rules and whether each one passed.
• The diff — the actual change.

The gate reads all four and writes one report file. One path, one truth. The moment two gates write two reports, people start quoting whichever one is green.

What the gate actually checks

Each check has a severity. Block means the verdict cannot be passed: true. Warn means the verdict still passes but the finding is printed on the report.

• Every acceptance command was actually run — block.
• Every acceptance command exited with code zero — block.
• No exit code is missing or null (that means a command silently never finished) — block.
• No write to a forbidden file — block.
• Every block-severity project rule passed — block.
• Files touched that were outside the allowed list but not forbidden — warn.

IMP

Important: A block finding cannot be cleared by the agent. Only a human can override it, and the override must record a reason and the person's user id in an audit log that lives in git. Otherwise you do not have an override policy, you only have theatre.

Deterministic means deterministic

Same artifacts in, same verdict out, every single time. No model is allowed inside the gate.

This confuses people, so keep the split clean. Deterministic checks answer "did the code solve the problem?" — tests, schemas, exit codes. Model-based review answers "is this readable, is it secure, is it in our style?" That second job belongs to the reviewer agent, not to the gate. Mix the two and neither signal means anything.

One gate is not enough

In real teams the gate is one layer in a stack: a pre-commit hook, a CI status check, a permission hook before risky tools run, then the pre-merge gate. Each layer is deterministic, so what one misses the next catches. The early hooks matter most, because they do not depend on the agent choosing to obey.

A worked example

Suppose Priya gives the agent a task: fix the coupon bug in pricing.py, and the acceptance command is pytest tests/test_pricing.py.

The agent comes back happy. The gate opens the feedback log. Two commands were run, pytest exited 0, good. Then it opens the scope report: the agent also edited .github/workflows/ci.yml. That file is on the forbidden list. Block.

Verdict: passed: false, one block finding. Priya reads it in ten seconds and sees the real story — the tests passed because the agent quietly weakened the CI config. Without the gate, she would have merged it.

IMPRemember: passing tests plus an untouched scope is "done". Passing tests alone is just a nice-sounding sentence.

Build it

"""A tiny verification gate. Deterministic in, deterministic out."""

# Each check returns a list of findings. A finding is (severity, code, detail).
BLOCK, WARN = "block", "warn"

def check_acceptance(feedback):
    """Every acceptance command must have run and exited zero."""
    out = []
    for cmd in feedback["expected_commands"]:
        record = feedback["runs"].get(cmd)
        if record is None:
            out.append((BLOCK, "cmd_not_run", cmd))
        elif record["exit_code"] is None:
            out.append((BLOCK, "exit_code_missing", cmd))
        elif record["exit_code"] != 0:
            out.append((BLOCK, "cmd_failed", f"{cmd} exited {record['exit_code']}"))
    return out

def check_scope(scope):
    """Forbidden writes block. Merely off-scope writes only warn."""
    out = []
    for path in scope["touched"]:
        if path in scope["forbidden"]:
            out.append((BLOCK, "forbidden_write", path))
        elif path not in scope["allowed"]:
            out.append((WARN, "off_scope_write", path))
    return out

def check_rules(rules):
    """Any block-severity project rule that failed stops the task."""
    return [(BLOCK, "rule_failed", name)
            for name, ok in rules.items() if not ok]

def verify(task_id, art, strict=False):
    """Pure function: same artifacts always give the same verdict."""
    findings = (check_acceptance(art["feedback"])
                + check_scope(art["scope"])
                + check_rules(art["rules"]))
    if strict:  # release branches: every warning becomes a hard fail
        findings = [(BLOCK, code, detail) for _sev, code, detail in findings]
    blocked = [f for f in findings if f[0] == BLOCK]
    return {"task_id": task_id, "passed": not blocked, "findings": findings}

def report(verdict):
    mark = "PASS" if verdict["passed"] else "FAIL"
    print(f"[{mark}] {verdict['task_id']}")
    for sev, code, detail in verdict["findings"]:
        print(f"   {sev:5} {code}: {detail}")

def artifacts(runs, touched, rules):
    return {
        "feedback": {"expected_commands": ["pytest", "ruff"], "runs": runs},
        "scope": {"allowed": ["pricing.py"], "forbidden": ["ci.yml"],
                  "touched": touched},
        "rules": rules,
    }

ok_runs = {"pytest": {"exit_code": 0}, "ruff": {"exit_code": 0}}

report(verify("T1-clean", artifacts(ok_runs, ["pricing.py"], {"no_secrets": True})))
report(verify("T2-scope-creep", artifacts(ok_runs, ["pricing.py", "ci.yml", "notes.md"],
                                          {"no_secrets": True})))
report(verify("T3-never-ran", artifacts({"pytest": {"exit_code": None}}, ["pricing.py"],
                                        {"no_secrets": False})))

Run it with python3 file.py. T1 prints PASS with no findings. T2 passes its tests, but ci.yml is a forbidden write, so the verdict flips to FAIL — while notes.md shows up only as a warn. T3 fails three ways: ruff never ran, pytest has a missing exit code, and a rule failed.

Now flip strict=True on T2 and watch the harmless warn turn into a blocker.

Where you will see this

• Claude Code and Cursor run a diff through tests and lint hooks before a change is treated as finished.
• GitHub branch protection refuses a merge until required status checks report green — that is a verification gate with a nice web page on top.
• Pre-commit hooks in most Python and JavaScript repos block a commit on formatting or secret-scanning failures.
• A Swiggy or Flipkart style support bot that issues refunds runs a deterministic policy check before the money actually moves.
• Any CI pipeline where the deploy job refuses to start unless the test job passed.

Common mistakes

• Letting a model decide whether the task passed. The model wrote the code, so it is the worst possible judge of the code. You lose the one honest signal you had.
• Trusting "tests passed" without the exit code. If nobody recorded it, the test may never have run. Treat a missing exit code as a failure, not as a blank.
• Making every finding a block. Then people start bypassing the gate entirely, and you are back to zero. Keep the block list small and truly non-negotiable.
• Letting the agent override its own blocks. An override without a human name and a written reason is just the agent marking its own paper again.
• No coverage floor. If nothing watches the coverage number, an agent can delete the failing test and every report stays comfortably green.

If they ask in an interview

Q: Why should a verification gate be deterministic instead of using an LLM judge?

A: Because the gate decides status, and status must be repeatable — the same artifacts have to give the same verdict every time. An LLM judge is useful for qualitative questions like readability or style, so that belongs in a separate reviewer step. Mixing the two makes it impossible to tell why something passed.

Q: What is the difference between a warn and a block finding?

A: A block finding prevents the verdict from passing and can only be cleared by a human override with a recorded reason and user id. A warn is printed on the report but still allows a pass. This split keeps the gate strict where it matters without making people route around it.

Q: One gate at merge time, or several gates?

A: Several, in layers — pre-commit hook, CI check, a permission hook before risky tools, then the pre-merge gate. Whatever slips past one layer gets caught by the next, and the early hooks do not depend on the agent cooperating.

Try these

1. Add a coverage_floor check to the gate: take a coverage percentage as an artifact and block if it is under 80. Decide where the floor number should be stored.
2. Add a check that blocks if the diff touches more than 15 files. Then argue with yourself about whether it should be a block or a warn.
3. Make the gate print a short Markdown summary alongside the verdict, with only the fields a reviewer actually needs. Defend what you left out.
4. Write an override function that takes a reason and a user id, refuses an empty reason, and appends one line to an audit list. Then confirm the agent has no way to call it.

Words, simply

Quick recap

• The agent never marks its own work done — a deterministic gate reads its artifacts and decides.
• Block findings stop the task and only a human with a recorded reason can clear them; warns just annotate.
• One gate, one report path, layered behind pre-commit and CI so nothing green is taken on trust.