OSWorld-Verified Explained: How We Measure Computer-Use Models

OSWorld-Verified asks a blunt question: can a model-agent system finish a task inside real software, or can it only explain what a person should click?

That distinction matters. A fluent answer in chat does not prove that an agent can find the right control, preserve state across applications, recover from a bad click, and leave the environment in the requested final state.

The benchmark is useful evidence for that gap. It is not a universal “best computer-use model” test, and its scores belong to complete evaluation setups rather than base models in isolation.

What the verified protocol measures

OSWorld runs tasks inside real desktop and web applications. An agent receives an instruction, observes the environment, chooses interface actions, and continues until it finishes or exhausts its budget. The evaluator then checks the resulting application state instead of grading a written explanation.

The verified result set repaired tasks and evaluation logic from the original release. The official protocol contains 369 tasks. The maintainers also accept a 361-task run when eight Google Drive tasks cannot be configured, but the excluded-task policy must be reported. That detail is small enough to disappear in a leaderboard label and large enough to matter when two close scores are compared.

The task mix covers office applications, browsers, file operations, system settings, and workflows that cross more than one interface. Success therefore depends on perception, planning, action generation, state tracking, and recovery working together.

The operator receipt is the sourced ledger.

The live OSWorld-Verified route currently exposes 27 sourced rows. The table below is generated from that ledger rather than copied into this article:

Table 1 Download CSV

Rank	Model	Score	Evidence
1	Claude Mythos 5	85%	Provider exact
2	Claude Fable 5	85%	Provider exact
3	Claude Opus 4.8	83.4%	Provider exact
4	Gemini 3.6 Flash	83%	Provider exact
5	Holo3-35B-A3B	82.6%	Benchmark exact

That count is the operator receipt for this refresh. Rows without displayable exact-source records remain withheld. Even within the sourced set, a provider_exact label means the provider published the result for a named setup; it does not mean every provider used the same agent scaffold.

Use the benchmark route for current rows, source badges, and model links. This article owns the explanation of what those rows can and cannot support.

A score belongs to the whole system

An OSWorld-Verified score can change when any of these change:

• the exact model or model variant
• a general-purpose, specialized, or framework-provided scaffold
• screenshot-only versus accessibility-tree or hybrid observations
• display resolution and environment image
• prompts, planning loop, and recovery logic
• action or step budget
• one attempt versus repeated attempts
• the 369-task or accepted 361-task protocol

This is why the result should be read as “this model-agent setup completed this share of the selected tasks under this protocol.” Shortening that to “the model scored X” is convenient, but it hides the machinery that produced X.

Before treating two rows as a race, open their source badges and match those fields. If the setup is not stated, the honest answer is that the comparison is incomplete.

OSWorld 2.0 is a separate benchmark generation

OSWorld 2.0 does not simply add more verified rows to the original protocol. It introduces 108 much longer workflows, a coherent user profile, authentic input artifacts, and dynamic events that can change the goal while an agent is working. Some tasks require the agent to ask the user for information.

The longer horizon changes what failure means. A system can lose state after dozens or hundreds of actions, mishandle an interruption, or complete most of a workflow without satisfying the final evaluator. Those are valuable signals, but they are not interchangeable with OSWorld-Verified task success.

BenchLM therefore keeps OSWorld-Verified and OSWorld 2.0 in separate lanes. The newer protocol is relevant evidence, not a reason to relabel historical results.

Where the benchmark stops.

The controlled environment makes repeatable evaluation possible. It also leaves out conditions that decide whether a production agent is safe and economical:

• current application versions and layout drift
• production authentication and account recovery
• organization-specific roles and permissions
• network failures, rate limits, and anti-bot systems
• destructive-action approval and audit requirements
• latency, token use, and operator supervision cost

A model can perform well in the benchmark and still fail your workflow because your browser extension, identity boundary, spreadsheet template, or approval policy is different. That is the honest limit of the public result.

How I would use it in a model decision

I would use OSWorld-Verified as a screening signal when the product depends on desktop or browser control. I would first eliminate rows whose setup cannot be matched, then check Terminal-Bench 2.0 for terminal-heavy work and BrowseComp for research workflows.

The final step is a small replay suite built from the real job: the same accounts, permissions, files, failure recovery, and approval boundaries the agent will face in production. Track task completion, destructive mistakes, human interventions, latency, and cost. A public score can narrow the field; it cannot approve deployment.

For the broader ranking context, see agentic models, computer-use recommendations, and how BenchLM handles benchmark confidence.

---

Reviewed by Glevd on July 15, 2026. Method claims use the official OSWorld project, implementation, paper, and OSWorld 2.0 project. Live rows come from the OSWorld-Verified source ledger.