Reading AI benchmarks without being taken for a ride

Every week some AI company announces that its new model has won a benchmark. A bar chart, one green bar taller than the grey ones, a percentage with one decimal place underneath. The problem is not that the numbers are lies. Most of the time they are even correct. The problem is that they say almost nothing about whether the model solves your task well. A benchmark measures a narrow, artificial task under laboratory conditions. Your work is not a narrow, artificial task. This playbook walks you through ten steps, and at the end of them you can look at a leaderboard without being dazzled. It is for you if you are starting to compare AI tools seriously and have no interest in falling for marketing graphics.

Step 1, what a benchmark actually measures

A benchmark is a collection of test tasks with a known correct answer. You give them to the model, count how often it gets them right, divide by the total, and there is your percentage. Sounds fair. The catch sits in the selection of the tasks. SWE-bench, for example, measures whether a model can fix real bugs in real software projects. MMLU measures multiple-choice knowledge across school subjects. HumanEval measures whether the model solves small programming exercises. Those are three completely different things. A model can lead on one and trail on the next. Before you believe a number, ask first: what exactly was measured there, and does it have anything to do with what I want to use the tool for.

Step 2, why SOTA is a marketing word

SOTA stands for state of the art and is supposed to mean "the best there currently is". In practice it means "the best on the one benchmark we picked for the chart". There are hundreds of benchmarks. A company presenting its model picks the three to five where it looks good and leaves out the rest. That is not even dishonest in the narrow sense, everybody does it. But it means that "SOTA" in a press release is worth about as much as "best restaurant on the street" when the company gets to decide which street is meant. Take the word for what it is: a signal that marketing is in the room.

Step 3, the contamination trap

This is the most important point and the one mentioned least. Models learn from enormous amounts of text from the internet. Many benchmarks sit openly on the internet, solutions included. If a model has already seen the test tasks and the correct answers during training, then the benchmark no longer checks whether it can solve the task, it checks whether it remembers the answer. This is called contamination or data leakage. A model with 92 percent on an old, famous benchmark can in truth be weaker than one with 80 percent on a fresh test that was never published. Rule of thumb: the older and more famous a benchmark, the more careful you should be with high scores.

Step 4, pass@1 versus best-of-N

Here hides a number trick that almost nobody explains. There are two ways to measure a result. With pass@1 the model gets one attempt, and it has to land. With pass@k or best-of-N it may try the task several times, and it counts as passed if even one of the attempts was right. The second number is always higher, often by a lot. A model that reaches 55 percent at pass@1 can suddenly land at 80 with best-of-10. Both are honest numbers, but they measure completely different things. In your daily work you usually get one attempt, so pass@1 is the relevant number. If a chart shows only the high number and does not say how often the model was allowed to try, be careful.

Step 5, who ran the test

It makes a difference whether a company tests its own model or whether an independent lab tests both models under the same conditions. When OpenAI says "GPT beats Claude" and Anthropic says "Claude beats GPT", then both are testing on home turf with their own settings. Self-measured numbers are not worthless, but they are advertising. Look for sources where someone without a stake did the measuring, ideally with the method disclosed. Independent leaderboards where you can see who tested how and whether it can be reproduced are worth gold. A chart in the vendor's announcement blog post is the start of the research, not the end.

Step 6, which version was measured

Models change constantly. A name like "Opus 4.8" or "GPT-5" stays the same while updates are rolled out behind it. A benchmark number from May may no longer hold for today's version, in either direction. When you read a number, look at the date and at the exact version label. That goes double for comparisons someone ran months ago and never updated. Comparisons like that haunt blog posts for a long time and get quoted as if they were still current. A value without a date and without a precise model version is a value you cannot place.

Step 7, the jump from 88 to 90 percent

When one model passes a benchmark with 90 percent and another with 88, that sounds like a difference. Often it is not. Many well known benchmarks are saturated, meaning the good models all sit in a narrow band right at the top. The last few percentage points then frequently consist of tasks that are ambiguous or plainly faulty even for humans. A two point lead in that region tells you little about practice. More interesting are benchmarks where there is still real spread, where the models genuinely pull apart. There you learn more about actual capability than from a race everybody finishes at 89 point something.

Step 8, the gap between benchmark and production

This is where most of the disappointment comes from. A model that scores highly on a particular class of tasks in the benchmark can still be unreliable in your real workflow. The benchmark runs in a clean environment with clearly stated tasks. Your reality has unclear instructions, messy inputs, long conversation histories and the demand that it works not once but every single time. It is entirely normal for a model to look strong in the lab and land significantly lower on your specific task in continuous operation. That is why the benchmark number is a pre-selection at best. It tells you which models make the shortlist, not which one wins.

Step 9, your own mini benchmark

This is the only test that really counts for you. Build yourself a small comparison out of ten to twenty tasks that look exactly like what you need every day. Real examples from your work, not invented ones. Give each task to two or three models, collect the answers and judge for yourself which one is usable. Note the results in a simple table. After an hour you know more about the models than any vendor chart could tell you, because you measured on your real tasks and not on school-subject multiple choice. Repeat it when a new version comes out. This is exactly the approach I also recommend in the playbook on choosing a model, where it is step eight.

Step 10, the three questions you always ask

When you see a benchmark number in future, wherever it is, ask yourself these three questions. First, what was measured, and does it have anything to do with my task. Second, who measured it, and did that person have an interest in the outcome. Third, can I follow it, is there a date, a version and a disclosed method. Answer yes three times and the number is a useful hint. If one question stays open, it is marketing and you can safely ignore it. Those three questions cost you ten seconds and protect you from most of the expensive wrong decisions that come from blindly trusting a pretty chart.

What comes next

Now that you know how to place numbers, the logical next step is translating them into a real choice. The playbook GPT-5 or Opus 4.8, which model shows you the tier thinking and your own mini eval on a concrete case. For the basics on why models behave so differently in the first place, read the lesson Models compared and the overview The model landscape 2026. And if you want to know what these models really cost in continuous operation rather than just what they can do, take a look at the playbook What AI really costs.