Agent eval in 60 minutes, how to find out if your agent is any good

The biggest blind-spot topic for solo founders who have built their first agent: nobody knows whether the thing is actually any good. You type in a few prompts, the agent delivers something, looks ok, done. And then you change the system prompt and it is still ok. And then two weeks later you notice that somewhere in between the quality dropped. But when? On which change?

That is exactly where agent eval comes in. You build yourself a small test set with inputs and an expectation. On every change it runs through and you see a pass rate. When it drops you know immediately that you broke something.

Over the next 60 minutes you build exactly that. No framework, no expensive tool subscription, just 20 test cases and a script that runs them. Enough for 80 percent of use cases.

1. Pick the agent you want to evaluate

Pick the agent you use most often. Not the coolest one, the most used one. If you built it two weeks ago from the sub-agent playbook and it runs every day, then that is your candidate.

Write down three sentences about the task. What is the input, what is the output, how do you tell that the output is good. Example: "Input is a bug description in plain text. Output is a GitHub issue title under 80 characters plus three labels from a fixed list. Good means: the title describes the problem in active voice, the labels are all from the list, no made-up label."

These three sentences are the spec. Without them you cannot evaluate. With them, everything that follows is just mechanics.

2. Create the eval set file

Make yourself a folder evals/ in the project and inside it a file cases.json. The format is a list of objects, each object is a test case with input and expected and an id.

[
  {
    "id": "001-simple-bug",
    "input": "Login klappt nicht mit Sonderzeichen im Passwort",
    "expected": {
      "title_max_chars": 80,
      "title_must_contain": ["login", "passwort"],
      "labels_subset_of": ["bug", "auth", "ux", "security"]
    }
  }
]

expected is not an exact output expectation but a list of conditions that the output has to satisfy. That is important. An exact string expectation would never pass because LLM output varies. You want conditions that are robust.

3. Collect at least twenty test cases

Twenty is the magic number. Under ten is statistics nonsense, over fifty you just fluff up the maintenance. Twenty cases that cover real use cases plus two or three ugly edge cases, that is the sweet spot.

Don't collect them from your head. Go into your chat history, into logs, into real tickets. Real inputs are gold. Made-up inputs are often too clean and don't cover edge cases. If you have used the agent for two weeks you already have twenty real inputs lying around somewhere.

Mark at least three of them explicitly as "was tricky back then" or "didn't work back then". These cases are your regression tests. When they fall out again you know that a change broke something.

4. Write one check function per condition

One small function per expectation type that returns a bool. For example title_max_chars checks whether the title is shorter than the allowed number. labels_subset_of checks whether all chosen labels are in the allowed list. must_contain checks whether certain keywords are in the output.

Keep it simple. JavaScript, Python, whatever you write anyway. Four or five such mini functions are enough to start.

function checkTitleMaxChars(output, max) {
  return output.title.length <= max
}

function checkLabelsSubsetOf(output, allowed) {
  return output.labels.every(l => allowed.includes(l))
}

Important: one function does exactly one thing. Not a "checkAll" with ten conditionals inside. Otherwise you can't tell later which condition failed.

5. Build the runner script

A loop over the cases. Per case you call the agent, get the output, run the check functions over it, collect a pass or fail per case.

const cases = JSON.parse(fs.readFileSync('evals/cases.json'))
const results = []
for (const c of cases) {
  const output = await callAgent(c.input)
  const checks = runChecks(output, c.expected)
  results.push({
    id: c.id,
    pass: checks.every(c => c.passed),
    failed: checks.filter(c => !c.passed).map(c => c.name)
  })
}
console.log(`Pass-Rate: ${results.filter(r => r.pass).length}/${results.length}`)

The script is deliberately dumb. No parallelization, no caching, no retry logic. You want to see whether your agent is good, not to optimize the runner. If the twenty cases take two minutes that is absolutely fine.

6. Do the first run and note the pass rate

Run the script once with the current agent. Note the pass rate. Probably not 20 out of 20. If it is, your eval cases are too lax.

A solo agent typically lands at twelve out of twenty. That is not bad. That is your baseline. From now on everything is make-it-better or make-it-worse.

Write the pass rate into a file evals/baseline.txt with the date and a short comment. "2026-04-30: 12/20, the failing ones are mainly the cases with unclear input." This comment is gold when in two weeks you want to know what didn't work back then.

7. Make a small change to the system prompt

Change exactly one thing. For example: into the system prompt you write a sentence about how the agent should handle unclear input. Not ten changes at once. One.

Run the eval again. Note the pass rate. If it goes up, great, keep the change. If it goes down, revert. If it stays the same: maybe the change helps in real use cases anyway, but you have no eval evidence that it does anything.

That is the core of the whole eval pattern: you make the change so the eval gets better, not because your gut feeling says it would be better. Gut feeling lies, eval pass rate lies harder to.

8. LLM-as-judge for fuzzy conditions

Some conditions you cannot check hard. "The title sounds natural" is one such example. There you call a second LLM, give it the output and the condition, and let it return a pass or fail with a reason.

async function llmJudge(output, criterion) {
  const prompt = `
Output: ${output}
Criterion: ${criterion}
Antworte mit PASS oder FAIL und einem Satz Begründung.
`
  const judgment = await callJudgeLLM(prompt)
  return judgment.startsWith('PASS')
}

Important: the judge LLM is not the same agent you are evaluating. Otherwise it makes everything look nice for itself. Take a different model, ideally a cheap one because this runs a hundred times per run. Haiku or Gemini Flash are good judges for such tasks.

LLM-as-judge is not perfect. Reckon with the judge verdict being roughly 80 percent correct. For a trend that is enough, for hard compliance decisions it is not.

9. Build the eval into your change workflow

From now on: every time you tinker with the agent, the eval runs before and after. You don't commit a change without knowing the pass rate. If you use Git, you write the pass rate into the commit message. "Pass: 15/20 (was 14/20)".

This has a psychological effect that should not be underestimated: you stop fiddling with the agent when it is at 18/20. You no longer do "let me just improve it a little more" and break something while doing it. The pass rate is your stop signal.

10. Let the cases grow when something surprises you

When the agent really screws up in the wild and the eval doesn't catch it, then your eval is the gap. Put the mess in as a new case. Over time the eval set gets robust and catches 90 percent of the problems before they land in production.

That is the most important rule: the eval set is alive. It grows with the agent. Throw out cases that no longer test anything, add new ones, and once a quarter check whether the old cases still fit the current task.

What comes next

Once you have the eval, the next step is multi-model comparison. Run the same twenty cases against Sonnet, Haiku, Gemini Flash and GPT-5-mini. You will be surprised how different the pass rates are and how often the cheapest model is not actually much worse than the most expensive one. For that we have Lesson 1.5 "Modelle vergleichen" and Recipe 5.x "Modell-Routing nach Pass-Rate". Also take a look at the playbook "Hooks gegen Halluzinationen" if you want to check the eval cases for robustness.

Source

• Anthropic Cookbook: Evaluations Patterns, github.com/anthropics/anthropic-cookbook (section evaluation/)
• Promptfoo Docs: promptfoo.dev/docs (open source, MIT license, good for your own eval setups)
• Inspect (UK AISI): inspect.ai-safety-institute.org.uk/docs (open-source eval framework, Python)