SWE-bench and GAIA

In plain words

A company cut-off of 8 CGPA tells you almost nothing about a student. Agent scores are the same. SWE-bench gives an agent a real bug report and decides only by running the project's own tests: the broken test must start passing and the working tests must not break. GAIA checks general tool use, AgentBench checks many environments, and none of them check your cost or your domain.

How it flows

A tiny example

patch = agent_fix(repo, issue_text)
apply(repo, patch)
fail_to_pass = run_tests(repo, bug_tests)
pass_to_pass = run_tests(repo, old_tests)
resolved = fail_to_pass and pass_to_pass
print(resolved)

Notice the and on the last line: fixing the bug is not enough, you must also break nothing.

What you will learn

• How agents are scored on real coding work, not on pretty answers.
• What SWE-bench, SWE-bench Verified, GAIA and AgentBench actually test.
• Why a benchmark score can be inflated, and how to spot it.
• What no benchmark tells you, and what you must measure yourself.

The problem, simply

Think about placement season in your college. Every company puts out a cut-off. "We take 8 CGPA and above." Simple number, easy to compare.

But you know the truth. In one branch the papers are easy and everybody gets 9. In another branch the same effort gets you 7.5. Two students with the same CGPA are not the same student.

Agent benchmarks have exactly this problem. Someone says "our agent scores 50% on SWE-bench." Sounds great. But 50% of what? Were the answers already sitting in the question paper? Did the checking even work properly? Does that test resemble the work you actually want done?

So before you quote any agent number, know three things: what is inside the benchmark, whether it is contaminated, and what it silently does not measure.

The idea

SWE-bench: fix a real bug, and the tests decide

SWE-bench is a coding benchmark. Plain line: it takes real bug reports from real open-source Python projects and asks the agent to fix them.

It has 2,294 issues taken from 12 popular Python repositories. The agent is given the codebase exactly as it was before the fix, plus the issue description written by a human. The agent must produce a patch, which is just a diff of the code changes.

Now the interesting part is the checking. No human gives marks, no other model judges. The harness applies your patch and runs the project's own test suite. Two gates must clear:

• FAIL_TO_PASS — tests that were failing before must now pass. This proves the bug is actually fixed.
• PASS_TO_PASS — tests that were already passing must still pass. This proves you did not break something else while fixing.

Suppose Priya's agent gets an issue: "date parser crashes on empty string." It adds a check and returns None. The empty-string test passes, so FAIL_TO_PASS is cleared. But now valid dates also return None and four old tests break. PASS_TO_PASS fails, so the task scores zero. No partial marks.

One early agent, SWE-agent, got about 12.5% here. Its main trick was not a smarter model but a better interface — clean file-editing commands and a search syntax the model could actually use. So the interface you give an agent moves the score as much as the model does.

SWE-bench Verified: the cleaned-up version

Some of those 2,294 issues are simply bad questions. Vague issue text, flaky tests, or a fix even a human engineer cannot pin down.

So a human-curated subset of 500 tasks exists, called SWE-bench Verified. It drops the ambiguous issues and the unreliable tests. This is the one to report when you claim your code agent works.

Contamination: the answer was in the question paper

Here is the uncomfortable part. Over 94% of SWE-bench issues are older than most models' training cutoff, so the model may have already seen these repos, issues and fixes during training.

A follow-up audit called SWE-bench+ looked at patches that passed. It found 32.67% had the solution leaking in the issue text itself — the fix was written right there in the bug report, so the agent just copied it. Another 31.08% were suspicious because the test coverage was too weak to really check the fix.

Warning

Warning: An agent showing 50% on plain SWE-bench can drop to around 35% on the cleaner version. If someone quotes only one number and never mentions Verified or contamination, be sceptical.

GAIA: easy for a human, hard for an AI

GAIA is a different flavour, not about code at all. It is a generalist benchmark — reasoning, browsing, tool use, and reading images and files together.

The design goal is stated plainly: conceptually simple for a human, hard for an AI. Humans score about 92%. An early GPT-4 with plugins scored about 15%.

It has 466 questions, 300 of them held back for a private leaderboard so nobody can tune against them. Three difficulty levels; Level 3 needs long tool chains across different kinds of data.

Think of Rahul being asked: "In this company's annual report, which quarter had the highest revenue, and what was the closing share price on the last day of that quarter?" A human with a browser does it in five minutes. An agent must search, download, read a table, then look up a price. Break any one link in that chain and the answer is wrong.

AgentBench: does it work outside one playground

AgentBench checks an agent across 8 environments, not one: code (Bash shell, databases, knowledge graphs), games (Alfworld, LTP), web (WebShop, Mind2Web), and open-ended generation.

All of them are multi-turn. The headline finding: what holds open-source models back against commercial ones is long-horizon reasoning, decision-making and plain instruction-following over many turns — not raw knowledge.

What none of these measure

• Real running cost — tokens burned and wall-clock time per task.
• Safety when someone is actively trying to break the agent.
• Your domain. A Python-repo benchmark says nothing about your fee-payment support bot.
• Tail failures. Benchmarks report an average; users remember the worst 1%.

IMPRemember: a benchmark is a proxy, not a promise. For your own product you must build your own small eval set.

Build it

A toy SWE-bench-style harness: three fake tasks, a scripted "agent" that proposes patches, and a runner that applies the two gates. No API calls, standard library only.

# A tiny SWE-bench-style harness. Tests decide, not opinions.

# Each task: the buggy function, plus tests split into two gates.
TASKS = [
    {
        "id": "T1-empty-string",
        "buggy": lambda s: len(s) // 0,          # crashes always
        "fail_to_pass": [("", 0)],               # must work now
        "pass_to_pass": [("abc", 3)],            # must not break
    },
    {
        "id": "T2-negative-fee",
        "buggy": lambda n: n * 100,
        "fail_to_pass": [(-2, 0)],               # negatives should clamp to 0
        "pass_to_pass": [(3, 300)],
    },
    {
        "id": "T3-rounding",
        "buggy": lambda p: int(p),
        "fail_to_pass": [(2.6, 3)],              # should round, not truncate
        "pass_to_pass": [(2.0, 2)],
    },
]

# The "agent". In real life a model writes these. Here they are scripted.
PATCHES = {
    "T1-empty-string": lambda s: len(s),                 # correct
    "T2-negative-fee": lambda n: max(n, 0) * 100,        # correct
    "T3-rounding": lambda p: int(p) + 1,                 # fixes the bug, breaks 2.0
}

def run_gate(fn, cases):
    """Return True only if every (input, expected) pair matches."""
    for arg, expected in cases:
        try:
            if fn(arg) != expected:
                return False
        except Exception:
            return False
    return True

resolved = 0
for task in TASKS:
    patch = PATCHES[task["id"]]
    f2p = run_gate(patch, task["fail_to_pass"])
    p2p = run_gate(patch, task["pass_to_pass"])
    ok = f2p and p2p                      # both gates, no partial marks
    resolved += 1 if ok else 0
    print(f"{task['id']:<18} FAIL_TO_PASS={f2p!s:<5} PASS_TO_PASS={p2p!s:<5} -> "
          f"{'RESOLVED' if ok else 'not resolved'}")

print(f"\nResolution rate: {resolved}/{len(TASKS)} = {100*resolved//len(TASKS)}%")

Look at task T3 in the output. The patch does fix the reported bug, so FAIL_TO_PASS is green. But it breaks a case that was already working, so PASS_TO_PASS is red and the task is not resolved.

That is the whole point. Fixing is not enough; not breaking counts equally. Your score is only the tasks where both gates went green.

Where you will see this

• Coding agents like Claude Code, Cursor and GitHub Copilot's agent mode publish SWE-bench Verified numbers at launch.
• Model release posts from big labs quote SWE-bench Verified as the headline coding score.
• Teams building browsing or "do my errand" assistants use GAIA-style tasks to check generalist ability.
• Internal agent teams copy the FAIL_TO_PASS idea to gate their own agent on their own repo before shipping.
• Support bots at Indian fintech and edtech companies skip public benchmarks and score against 200 real past tickets instead.

Common mistakes

• Quoting one number and stopping. "We hit 50%" hides cost, steps and contamination. Say which split, and pair the score with tokens and time per task.
• Reporting plain SWE-bench instead of Verified. The unfiltered set has vague issues and flaky tests, so part of your number is noise.
• Ignoring contamination. If the model saw the repo and the fix in training, a good score proves memory, not skill.
• Tuning towards the benchmark. Once the benchmark becomes the target it stops being a measurement — score up, product flat.
• Using a code benchmark for a non-code agent. SWE-bench says nothing about whether your support bot handles a refund request.

If they ask in an interview

Q: How is an agent evaluated on SWE-bench?

A: The agent gets the repository at the pre-fix commit plus the issue text, and produces a patch. The harness applies it and runs the repo's own tests. It counts as resolved only if the previously failing tests now pass and the previously passing ones still pass.

Q: Why would you trust SWE-bench Verified more than the full SWE-bench?

A: Verified is a 500-task subset humans went through by hand to remove ambiguous issues and unreliable tests. On the full set some tasks are unsolvable or badly checked, so part of the score is noise. Verified is cleaner, though still not free of contamination.

Q: What is benchmark contamination and how would you check for it?

A: It means the tasks or their solutions were already in the model's training data, so the score reflects memory, not reasoning. I would check whether the tasks predate the model cutoff, scan the issue text for the fix being spelled out, and confirm the tests are strong enough to catch a wrong patch. The real proof is a fresh private task set the model cannot have seen.

Try these

1. Run the harness, then add a fourth task where the patch clears PASS_TO_PASS but fails FAIL_TO_PASS. Confirm it still scores zero.
2. Take a small Python project of yours with a known bug. Write two failing tests for it and list three tests that already pass. That is your own mini benchmark.
3. Add a step counter to the harness and print average patch attempts per resolved task next to the resolution rate.
4. Write a leakage check: flag a task if any line of the patch already appears inside the issue text.

Words, simply

Quick recap

• SWE-bench scores a code agent by running the repo's real tests: the bug test must flip to passing and nothing already passing may break.
• Report SWE-bench Verified and mention contamination — a large share of "successful" patches had the answer sitting in the issue text.
• GAIA tests generalist tool use, AgentBench tests many environments, but none of them test your cost, safety or domain. Build a small eval of your own.