Agent benchmarks: what SWE-bench, τ-bench and GAIA actually measure
The agent benchmarks worth knowing — SWE-bench Verified, τ-bench with its pass^k reliability metric, GAIA's human-vs-model gap — where leaderboard numbers mislead, and a reading checklist for the next score a vendor shows you.
Definition
Agent benchmarks are standardised task suites — SWE-bench for software issues, τ-bench for tool-and-user interaction, GAIA for assistant tasks — used to compare agents and models. They measure performance on the benchmark's tasks under the benchmark's harness, which correlates with, but never guarantees, performance on yours.
Benchmarks are how the field compares agents — and how vendors sell them, which is why reading them well is an operating skill, not trivia. A benchmark score is a real measurement of a narrow thing: this model, in this harness, on these tasks, this many attempts. Confuse it with “how good the agent is” and you will ship the wrong system; read it precisely and it saves you weeks of shortlisting.
Three suites cover most of what production teams need to know, and each teaches a different lesson about measurement itself.
The three that matter, and their honest numbers
| Benchmark | Task | The number to remember |
|---|---|---|
| SWE-bench / Verified | Resolve real GitHub issues in real repositories | Verified = 500 tasks human-confirmed solvable, released Aug 2024 with OpenAI’s Preparedness team |
| τ-bench | Tool use + simulated users + policy rules (retail, airline) | gpt-4o under 50% single-attempt; pass^8 under 25% in retail |
| GAIA | Real-world assistant questions needing browsing and tools | Humans 92%, GPT-4 with plugins 15% at release — 466 questions “conceptually simple for humans” |
Each row carries a caution. SWE-bench needed a Verified subset because a chunk of the original tasks were underspecified or unsolvable. The most-cited agent benchmark shipped for months with tasks no agent could pass, and nobody noticed until human annotators screened all of them. τ-bench (Yao et al., 2024) exists because single-attempt scores flatter agents: its pass^k metric requires the same task to succeed k times in a row, and reliability collapses quadratically faster than capability. GAIA (Mialon et al., 2023) inverts the usual difficulty story: its questions are easy for people and were nearly impossible for tooled GPT-4. “Superhuman on exams” and “reliable assistant” are different axes entirely.
One structural note as of late 2025: SWE-bench’s maintainers restricted Verified and Multilingual submissions to teams with open-source methods and a research publication. That is a benchmark defending itself against becoming a marketing surface — and a hint about how much weight unverifiable leaderboard claims were carrying.
Where the numbers mislead
Four failure modes account for most benchmark misreadings:
- Contamination. Public tasks age into training data. A score on tasks the model may have seen is a memory test, and you cannot tell from the outside how much of one.
- Harness variance. “Model X scores Y” hides the scaffold: prompts, tools, retry policy, context management. The same model moves materially between scaffolds, which is why a score without the harness description is not reproducible — or comparable.
- Attempt accounting. pass@k versus pass^k, best-of-n versus first-try. An unattended production agent lives at pass^k; most headlines are pass@1 at best.
- Goodhart’s law. Once a leaderboard drives purchasing, optimising for the leaderboard diverges from optimising for the job. The benchmark stays fixed; your task drifts away from it.
The reading checklist
Before any leaderboard number changes a decision, make it answer six questions. This is the checklist I keep next to procurement conversations — copy it:
1. Which variant, exactly? (SWE-bench full ≠ Verified ≠ Lite)
2. What harness? (scaffold, tools, context policy — published or vibes?)
3. What attempt policy? (pass@1? best-of-8? pass^k?)
4. Could the tasks be in training data? (public since when? any private split?)
5. Who ran it? (self-reported vendor run vs leaderboard-verified)
6. What would this look like on MY task? (if no answer: run 20 of your own cases)Question six is the exit from the whole game. A benchmark shortlists two or three candidates. Twenty representative cases from your own workload, scored the way your evals score them, decide the winner. Unlike the leaderboard, they keep deciding after every model update, as regression tests.
The benchmark nobody publishes is the one that governs you
This portfolio runs on that principle in miniature. No public leaderboard measures “writes publishable trilingual articles without AI-slop or invented facts”. So the gate here is a bespoke check: it scans all 36 articles per language for banned phrases, missing definitions and sentence bloat, and blocks deploy on any error. It is a tiny benchmark with an n of one team, and it has caught real regressions no SWE-bench score would surface. Resolving GitHub issues and refusing to fabricate a statistic are unrelated skills. That is the general shape of the answer: public benchmarks to choose a model, a private suite to run your life, and production metrics to check that both were right. Scoring the open-ended parts of that private suite is its own craft — covered in LLM-as-a-judge.
Frequently asked questions
Do benchmark scores predict production performance?
They rank capability; they do not predict your outcome rate. A benchmark fixes tasks, tools and a harness that are not yours, and public tasks leak into training data over time. τ-bench made the gap measurable: agents that look strong on single attempts drop below 25% when the same task must succeed eight times in a row. Use scores to shortlist, then measure on your own cases.
Which benchmark is closest to my use case?
Match the task shape. Coding agents: SWE-bench Verified, because its 500 tasks are human-confirmed solvable. Customer-facing agents with tools and policies: τ-bench, whose simulated users and pass^k metric test rule-following and consistency. Research or browsing assistants: GAIA. If none matches, that is your answer — your evals are the benchmark that matters.
What is pass^k and why does it matter more than pass@k?
pass@k asks whether at least one of k attempts succeeds — generous to lucky runs. τ-bench's pass^k asks whether all k attempts succeed, which is what an unattended agent actually owes you. The two tell different stories about the same system: gpt-4o solved under half of τ-bench retail tasks once, and under a quarter reliably across eight tries.
Should I trust a vendor's benchmark claim?
Trust it as far as the harness is published. The same model moves several points between scaffolds, so a score without the agent configuration, tool set and attempt policy is marketing. SWE-bench's maintainers now accept Verified submissions only from teams with open methods and a research publication — a sign of how much unverifiable claims had muddied the water.