WebArena and OSWorld

In plain words

When your friend books an IRCTC ticket for you, you do not grade the words he typed on each screen. You check whether a real ticket exists. WebArena and OSWorld grade browser and desktop agents the same way: run the task, then look at the final state of the app or the computer. And they also count steps, because an agent that wanders takes three times the cost for the same result.

How it flows

A tiny example

app = new_app()
for action, arg in agent_actions(task):
    app = step(app, action, arg)
passed = task.check(app)          # look at state, not words
efficiency = len(agent_actions(task)) / len(task.gold)
print(passed, efficiency)

Notice that the agent's own text is never read; the score comes from the app state, and efficiency compares the agent's steps to the human's shortest path.

What you will learn

• What WebArena and OSWorld are, and why both exist.
• Why these benchmarks check the final state instead of reading the agent's answer.
• The two reasons desktop agents fail: not finding the button, and not knowing the app.
• Why counting steps matters as much as counting successes.

The problem, simply

Suppose you tell your junior, "Book a Chennai to Bengaluru ticket on IRCTC for next Friday." He does not reply with an essay about railways. He opens the site, logs in, types the stations, picks the date, selects a train, fills passenger details, pays by UPI, and shows you the PNR.

Now think about how you would grade him. You would not read every screen he typed on. You would check one thing: is there a valid ticket or not?

Agents that drive a browser or a desktop have the same problem. Grading a one-shot answer is easy. Grading twenty screens of clicking is a different kind of test.

WebArena and OSWorld are two benchmarks built for this. A benchmark is just a fixed set of tasks plus a fixed way to score them, so different people can compare fairly. WebArena tests agents inside a web browser. OSWorld tests agents on a full computer desktop.

The idea

WebArena: a small internet you can restart

WebArena gives the agent four websites that the benchmark runs on its own machines: a shopping site, a discussion forum, a GitLab-style developer tool, and a business content system for managing pages. It also gives small helpers like a map, a calculator and a scratchpad. There are 812 tasks in total.

The important word is self-hosted. The sites run locally at a pinned version, so they behave the same today and next month. Test against the real Flipkart instead, and prices and layouts change every week, so your scores mean nothing.

Scoring is execution-based. That means the grader looks at the final state of the app, not at the agent's words. Was the order actually placed? Was the issue actually closed? Was the page actually updated?

IMP

Note: Execution-based grading is why these benchmarks are trusted. The agent cannot talk its way to a pass.

At release time, the best agent finished about 14.41% of WebArena tasks. Humans doing the same tasks finished about 78.24%. That gap was the whole point of publishing it.

There are two follow-ups worth knowing by name. VisualWebArena adds tasks where you must actually look at images on the page to succeed. TheAgentCompany adds a terminal and coding work, so it feels closer to a real remote job.

OSWorld: the whole computer, not just the browser

OSWorld goes one level wider. It has 369 tasks on real Ubuntu, Windows and macOS machines. The agent controls the keyboard and mouse freely, on real applications.

Here is the key design choice. The only thing the agent sees is a screenshot of the screen at 1920x1080. No hidden list of buttons, no accessibility API (that is the operating system's built-in list of on-screen elements meant for screen readers).

Why be so strict? Because that is the real constraint in production. A shipped desktop agent gets pixels. If your benchmark hands the agent a clean list of buttons, you have quietly deleted the hardest part of the job.

The two ways desktop agents fail

GUI grounding. The agent knows it must click "Save As", but it cannot point at the right pixels. Mapping what it sees to where to click is genuinely hard on a big screen.

Operational knowledge. The agent does not know which menu holds that setting, or which shortcut opens it. You built this knowledge over years of using laptops. The model did not.

At release, the best model scored about 12.24% on OSWorld against about 72.36% for humans.

Two follow-ups you should know

OSWorld-G is a 564-sample set that tests only grounding, along with a training set. Separating grounding from planning lets you find out which half is broken instead of guessing.

OSWorld-Human adds gold trajectories: expert humans recorded the minimum action sequence for each task. With that, you can measure something success rate hides. Top agents take roughly 1.4 to 2.7 times more steps than needed.

IMPRemember: two agents can both pass 60% of tasks while one costs you three times more, because it wandered. Success rate alone will never tell you that.

A worked example

Suppose Priya is building an agent that books cab rides on an internal portal. She writes a task: "Book a cab from the hostel gate to the airport at 6 AM."

If she grades by reading the agent's final message, the agent can reply "Booked successfully" and pass while nothing happened. If she grades by checking the bookings table for a row with the right pickup, time and destination, the agent must actually do the work.

Then she records her own click path: 6 actions. The agent took 17. Same pass, very different bill.

Build it

"""Toy web-agent harness: execution-based scoring + trajectory efficiency."""

# The "app": a tiny shopping site the agent can act on.
def new_app():
    return {"cart": [], "order_placed": False, "page": "home"}

def step(app, action, arg=None):
    """One agent action changes app state, like a real click would."""
    if action == "open":
        app["page"] = arg
    elif action == "add_to_cart" and app["page"] == "items":
        app["cart"].append(arg)
    elif action == "checkout" and app["cart"]:
        app["order_placed"] = True
    return app

# Tasks: what the agent must do, how to check it, and the human's shortest path.
TASKS = [
    {
        "name": "Buy one notebook",
        "check": lambda a: a["order_placed"] and a["cart"] == ["notebook"],
        "gold": [("open", "items"), ("add_to_cart", "notebook"), ("checkout", None)],
        # The agent wanders: opens home twice before doing real work.
        "agent": [("open", "home"), ("open", "items"), ("open", "items"),
                  ("add_to_cart", "notebook"), ("checkout", None)],
    },
    {
        "name": "Buy a pen and a bag",
        "check": lambda a: a["order_placed"] and set(a["cart"]) == {"pen", "bag"},
        "gold": [("open", "items"), ("add_to_cart", "pen"),
                 ("add_to_cart", "bag"), ("checkout", None)],
        # The agent forgets the bag, so the final state check fails.
        "agent": [("open", "items"), ("add_to_cart", "pen"), ("checkout", None)],
    },
]

passed = 0
for task in TASKS:
    app = new_app()
    for action, arg in task["agent"]:
        app = step(app, action, arg)
    check = task["check"]            # execution-based: look at state, not words
    ok = check(app)
    passed += ok
    ratio = len(task["agent"]) / len(task["gold"])
    print(f"{task['name']:<20} pass={str(ok):<5} steps={len(task['agent'])} "
          f"gold={len(task['gold'])} efficiency={ratio:.2f}x")

print(f"\nSuccess rate: {passed}/{len(TASKS)} = {100 * passed / len(TASKS):.0f}%")

Look at two things in the output. First, the pass or fail comes from check, which reads the cart and the order flag, never the agent's claims. Second, the efficiency number: the first task passes but takes 1.67 times the human path, so it is a pass you should still be unhappy about.

Where you will see this

• Computer-use agents from Anthropic, OpenAI and Google are all measured on OSWorld-style desktop tasks.
• Browser agents that fill forms or place orders for you are measured on WebArena-style flows.
• Coding agents like Claude Code, Cursor and GitHub Copilot are graded the same way: did the tests pass, not did the answer sound right.
• Customer-support bots inside a company portal: the real check is whether the refund row appeared, not whether the bot said "done".
• Any internal team that records its top twenty user flows and replays them against the agent every week.

Common mistakes

• Grading the agent's final message. The model will happily say "task completed" when nothing changed. Always check the app state instead.
• Testing a DOM-reading agent on a screenshot benchmark. If your agent gets a clean element list, it skipped the grounding problem entirely, so the score means nothing.
• Reporting only success rate. You will not notice the agent taking twice or thrice the needed steps, and steps are money and latency.
• Upgrading the pinned test apps casually. WebArena pins app versions on purpose. Change a version without re-checking the tasks and your old scores are no longer comparable.
• Assuming a benchmark score transfers to your product. Your portal is not their shopping site. Build a small gold-trajectory set for your own top tasks.

If they ask in an interview

Q: What is execution-based evaluation and why do agent benchmarks use it?

A: Instead of comparing the agent's text to a reference answer, you check the final state of the environment: was the order placed, was the issue closed. It is used because a long-horizon agent can produce a confident summary without having done anything, and only state tells you the truth.

Q: Why does OSWorld give the agent only screenshots?

A: Because a real desktop agent in production only gets pixels. Handing it an accessibility API would hide the GUI grounding problem, which is one of the two biggest failure modes the benchmark is meant to expose.

Q: Two agents both score 60%. How would you pick one?

A: Compare trajectory efficiency, which is agent steps divided by the human gold path. Studies on desktop agents show top agents take roughly 1.4 to 2.7 times the necessary steps, and those extra steps are real cost, latency and extra chances to break something.

Try these

• Add a third task to the toy harness above, along with its gold path, and see how the success rate changes.
• Add a second toy app, a forum with open_thread and post_reply, and write three tasks with gold paths for it.
• Add a distractor action that no gold path ever uses, then write an agent path that gets tempted by it. Watch the efficiency number blow up.
• Take any small flow on a site you use daily, write down your own click path, and count the steps. That is your gold trajectory for that task.

Words, simply

Quick recap

• WebArena tests browser agents on pinned self-hosted sites; OSWorld tests desktop agents on real screenshots.
• Grade on final state, never on what the agent says it did.
• Success rate is half the story; measure steps against a human gold path too.