An eval suite is the test suite for that behavior. Most teams know they need one and put it off, because it sounds like a research project. It does not have to be. This guide walks through a small, useful first version you can build in an afternoon, then shows how to grow it.

The vocabulary, in one minute

Anthropic's engineering post Demystifying evals for AI agents uses a set of terms that are worth adopting, because they keep conversations about evals precise:

• Task: one test case, with defined inputs and a definition of success.
• Trial: one attempt at a task. You run several, because agent output varies.
• Grader: the logic that scores a trial. A task can have more than one.
• Transcript: the full record of a trial: messages, tool calls, intermediate results.
• Outcome: the state of the world when the trial ends.

That last distinction matters more than it looks. The outcome is whether the reservation exists in the database, not whether the agent said "I booked it."

Step 1: Collect 20 to 50 tasks from real failures

You do not need hundreds of tasks to start. Early on, the differences between versions of your agent are large, and a small set catches them. The Anthropic post suggests 20 to 50 simple tasks drawn from real failures as a good starting point, and warns that evals get harder to build the longer you wait.

Good sources for your first tasks:

• Bugs you already fixed. Each one is a regression test waiting to be written.
• Support tickets and user complaints.
• The manual checks you run before every release. If you type the same three prompts into a chat window every time you ship, those are tasks.
• Edge cases you argued about in code review.

Write each task down with its input and a plain-language statement of what success looks like. If two people on your team would disagree about whether a given result passes, the task is not ready yet.

Step 2: Define success as an outcome you can check

For every task, ask what you could inspect after the run to know it worked. Some examples:

Checking the environment instead of the agent's claims removes a whole class of false passes. Agents are good at sounding finished.

It also helps to keep each trial isolated. Reset the database, the file system or the mock services between trials, so one run cannot leak state into the next.

Step 3: Pick graders, cheapest first

There are three kinds of graders, and each has a place:

Start with code graders wherever the outcome is checkable. Add a model grader only for the parts of a task a script cannot judge, and give it a narrow rubric: one question per dimension, with an explicit "can't tell" option so it does not guess. Then spot-check a sample of its verdicts by hand. A model grader you have never compared against human judgment is a second untested system.

Grade what the agent produced, not the exact steps it took. A rule like "must call search then fetch then summarize" fails agents that find a better route. If you care about efficiency, measure it separately (tool calls, tokens, turns) instead of hard-coding a path.

Step 4: Run several trials and measure consistency

One run per task tells you what happened once. Run each task several times and you can measure two different things:

• pass@k: the chance that at least one of k attempts succeeds. Useful when one good answer is enough, such as generating candidate code you will test anyway.
• pass^k: the chance that all k attempts succeed. Useful for anything customer-facing, where every run has to work.

The gap between them is large. An agent that succeeds 75 percent of the time has a pass^3 of about 42 percent (0.75 × 0.75 × 0.75). Three customers in a row will all get a good result less than half the time. The pass^k metric was popularized by the τ-bench paper for exactly this reason: reliability across repeated runs is what users feel.

With n trials per task and c successes, you can estimate both from your data. The pass@k estimator comes from the Codex paper; the pass^k one uses the same counting idea.

from math import comb

def pass_at_k(n: int, c: int, k: int) -> float:
    """Chance that at least one of k sampled trials passes."""
    if n - c < k:
        return 1.0
    return 1.0 - comb(n - c, k) / comb(n, k)

def pass_hat_k(n: int, c: int, k: int) -> float:
    """Chance that all k sampled trials pass."""
    if c < k:
        return 0.0
    return comb(c, k) / comb(n, k)

# 8 trials, 6 passed
print(round(pass_at_k(8, 6, 3), 3))   # 1.0
print(round(pass_hat_k(8, 6, 3), 3))  # 0.357

Report both. A change that raises pass@k while lowering pass^k made your agent more capable and less dependable at the same time.

Step 5: A minimal harness

You do not need a framework for the first version. Here is the whole loop in plain Python. Replace run_agent with a call into your own agent, and write graders that inspect whatever your agent changes.

import json
from dataclasses import dataclass, field
from typing import Callable

@dataclass
class Task:
    id: str
    prompt: str
    graders: list[Callable[[dict], bool]]
    setup: Callable[[], dict] = field(default=lambda: {})

def run_agent(prompt: str, env: dict) -> dict:
    """Call your agent here. Return the transcript plus anything you need to grade."""
    raise NotImplementedError

def run_suite(tasks: list[Task], trials: int = 5) -> dict:
    results = {}
    for task in tasks:
        passes = 0
        transcripts = []
        for _ in range(trials):
            env = task.setup()                 # fresh state for every trial
            transcript = run_agent(task.prompt, env)
            ok = all(grade(env) for grade in task.graders)
            passes += ok
            transcripts.append({"passed": ok, "transcript": transcript})
        results[task.id] = {"passes": passes, "trials": trials, "runs": transcripts}
    return results

if __name__ == "__main__":
    results = run_suite(TASKS)
    with open("eval_results.json", "w") as f:
        json.dump(results, f, indent=2, default=str)
    for task_id, r in results.items():
        print(f"{task_id}: {r['passes']}/{r['trials']}")

A grader in this setup is just a function that looks at the environment after the run:

def meeting_was_booked(env: dict) -> bool:
    events = env["calendar"].events_on("2026-11-12")
    return any(
        e.start.hour == 14 and {"dana@example.com"} <= set(e.attendees)
        for e in events
    )

Save every transcript. You will need them in the next step.

Step 6: Read the transcripts

This is the step teams skip, and it is the one that makes the numbers trustworthy. Before you believe a score, open a handful of passing and failing transcripts and check two things:

1. Did failures fail for real? Sometimes the agent did the right thing and the grader rejected it, for example because a date was formatted differently. That is a grader bug.
2. Did passes pass for real? Sometimes the agent got lucky or the grader was too loose.

A task with a 0 percent pass rate across many trials is usually a broken task, not a hopeless agent. A good habit is to write a reference solution for each task, a run you know is correct, and confirm your graders accept it.

Step 7: Split capability and regression suites

As the suite grows, keep two lists:

• Capability evals ask what the agent can do. They should start with a low pass rate, so you have room to improve.
• Regression evals ask whether the agent still does what it used to. They should sit near 100 percent, and any drop blocks a release.

When a capability task passes reliably, move it to the regression suite. Over time, your regression suite becomes a record of every problem you have solved, and your capability suite stays pointed at what you have not.

Common mistakes

• Waiting for a big suite. Twenty real tasks this week beat three hundred synthetic ones next quarter.
• Grading the agent's words. Check the world, not the claim.
• One trial per task. You will mistake luck for progress.
• Only positive cases. Include tasks where the right move is to refuse, ask a question or do nothing.
• Never reading transcripts. A score you have not inspected is a guess.

Where to go next

Once the basic loop works, the most useful additions are cost and latency tracking per task, a model grader for open-ended answers, and running the suite in CI on every prompt or tool change. If your agent's failures are mostly about picking the wrong tool, the next post covers that directly: why your agent calls the wrong tool.

Sources

• Anthropic, Demystifying evals for AI agents (2026)
• Yao et al., τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024)
• Chen et al., Evaluating Large Language Models Trained on Code (2021)

Usually the problem is the tools. A model only knows what a tool does from its name, its description, its parameters and what it returns. If those are unclear to someone who has never seen your codebase, they are unclear to the model too. Anthropic's engineers made the same point in Building effective agents: on one of their coding agents, they spent more time improving the tools than the overall prompt.

Here is a checklist of the causes that come up most often, roughly in the order worth checking them.

First, classify the failure

Pull a few failing transcripts and sort each failure into one of these buckets:

1. Wrong tool: the agent picked a tool that cannot do the job.
2. Right tool, wrong arguments: the call was sensible but the inputs were malformed or mismatched.
3. Right call, misread result: the tool worked but the agent drew the wrong conclusion from its output.
4. Stuck: the agent repeated a call, gave up, or stopped before finishing.

The bucket points to the fix. Wrong-tool failures usually mean the tool set or descriptions are confusing. Wrong-argument failures point at parameter design. Misread results point at response format. Getting stuck often traces back to error messages.

Cause 1: Too many overlapping tools

A common pattern is to wrap every API endpoint as its own tool. The agent then sees list_users, get_user, search_users and find_user_by_email, and has to guess which one you meant.

Anthropic's post Writing effective tools for AI agents recommends building fewer tools around whole workflows instead. Their examples:

• Replace list_users, list_events and create_event with one schedule_event tool that finds availability and books the slot.
• Replace read_logs with search_logs, which returns only the relevant lines.
• Replace get_customer_by_id, list_transactions and list_notes with get_customer_context.

Fix: list your tools and, for each pair, ask whether a new teammate could tell from the names alone which one to use. Merge any pair where the answer is no. Each tool should have one clear purpose that no other tool shares.

Cause 2: Names that do not separate

Once an agent has tools from several services, names collide. Is search for the wiki, the ticket tracker or the codebase?

Fix: namespace by service and resource, for example jira_issues_search and confluence_pages_search. The Anthropic tools post notes that prefix versus suffix naming had measurable effects in their evals and that the best choice varied by model, so if you have an eval suite, test both.

Cause 3: Descriptions written for people who already know the system

A description like "Gets account data" makes sense to the person who wrote the endpoint. It tells the model nothing about what an account is, what data comes back or when to prefer another tool.

Fix: write the description the way you would explain the tool to a new hire. Include:

• What it does and what it returns, in one or two sentences.
• When to use it, and when to use a different tool instead.
• Any terms of art, formats or limits ("dates are ISO 8601", "returns at most 50 results").
• One short example call.

Before:

{
  "name": "get_account",
  "description": "Gets account data.",
  "input_schema": {
    "type": "object",
    "properties": { "account": { "type": "string" } }
  }
}

After:

{
  "name": "billing_account_get",
  "description": "Look up one billing account and return its plan, status, renewal date and the last 5 invoices. Use this when you already know the account ID. To find an account from a company name or email, call billing_accounts_search first.",
  "input_schema": {
    "type": "object",
    "properties": {
      "account_id": {
        "type": "string",
        "description": "Billing account ID, formatted like 'acct_1234'. Not the company name."
      }
    },
    "required": ["account_id"]
  }
}

The second version answers the questions the model would otherwise guess at. The Anthropic post reports that small refinements to tool descriptions produced large gains on their own benchmarks.

Cause 4: Parameters that invite mistakes

user could be a name, an email or an ID. date could be a day, a timestamp or "next Tuesday". Every ambiguous parameter is a coin flip.

Fix: make the wrong input hard to send.

• Name parameters by what they hold: user_id, not user.
• Use enums for any field with a fixed set of values.
• Mark required fields as required.
• Choose formats that leave no room for error.

The Building effective agents post has a good example of that last point. Their coding agent made mistakes with relative file paths after it moved out of the root directory. They changed the tool to require absolute paths, and the errors went away. They call this poka-yoke, a term from manufacturing for designs that prevent errors instead of detecting them.

Cause 5: Responses full of noise

If a tool returns a 3,000-token JSON blob of internal fields, the agent has to dig through all of it to find the one value it needs, and that blob sits in context for the rest of the conversation.

Fix: return what the agent needs to make its next decision, in a form it can use.

• Prefer human-readable identifiers. The Anthropic tools post found that resolving UUIDs to names or short numeric IDs improved retrieval precision and reduced hallucinations.
• Drop low-level fields the agent will never act on, like MIME types or thumbnail URLs.
• Offer a response_format parameter with concise and detailed options. In their example, the concise response used about a third of the tokens of the detailed one.
• Paginate or truncate large results, and say so in the response: "Showing 20 of 312 results. Narrow the query or request the next page." Claude Code, for reference, caps tool responses at 25,000 tokens by default.

Cause 6: Errors the agent cannot act on

When a call fails with Error 400 or a stack trace, the agent has nothing to work with. It retries the same call, or it gives up.

Fix: write error messages that tell the agent what to do next.

Bad:   ValidationError: invalid value
Good:  start_date must be YYYY-MM-DD, for example 2026-11-12. You sent "next Tuesday".
       Resolve relative dates before calling this tool.

A good error message turns a dead end into a one-step correction.

Cause 7: The tool set changes between turns

Some agents add and remove tools as the conversation moves, to keep the list short. This can confuse the model when earlier turns refer to tools that no longer exist, and it also defeats prompt caching, since tool definitions sit at the very start of the prompt. If you need to restrict which tools can run at a given step, both the Claude and OpenAI APIs offer a tool_choice setting, and OpenAI's caching guide suggests restricting tools that way instead of editing the list. More on that in prompt caching for AI agents.

How to know you fixed it

Guessing is how tool problems come back. Build a small set of tasks that each require a specific tool or sequence, run them a few times, and track:

• Pass rate on the tasks.
• Number of tool calls per task.
• Number of tool errors per task.
• Tokens per task.

Then change one thing at a time: a description, a name, a merged tool. If the numbers move the right way, keep the change. If you have not set up evals yet, this starter guide covers the basics.

Quick checklist

• Can a new teammate tell every tool apart by name and description alone?
• Does each description say when not to use the tool?
• Is every parameter named for what it holds, with enums and required fields set?
• Do responses use readable identifiers and skip internal fields?
• Are large results paginated, with a note saying so?
• Do errors explain how to fix the call?
• Is the tool list stable across the conversation?

Sources

• Anthropic, Writing effective tools for AI agents (2025)
• Anthropic, Building effective agents (2024)
• OpenAI, Prompt caching guide

Prompt caching fixes most of that. Both Anthropic and OpenAI let you reuse the processed form of a prompt prefix you sent recently, at a fraction of the normal input price. The catch is that the cache only hits if your prompt is built the right way, and agents are very good at accidentally breaking it.

This post covers how caching works, how to structure an agent so it hits, and how to check that it is working.

How prompt caching works

The idea is the same on both platforms. When a request starts with exactly the same tokens as a recent request, the provider can skip reprocessing that shared beginning and charge you less for it.

Two rules follow from that:

1. Only the prefix counts. The cache matches from the start of the prompt and stops at the first difference. A changed character near the top makes everything after it a miss.
2. Matches are exact. Anthropic's documentation says cache hits require 100 percent identical prompt segments. "Almost the same" is a miss.

So the whole game is ordering: put what never changes first, and what changes every turn last.

The order that works

For an agent, that usually means:

1. Tool definitions. Same on every turn.
2. System prompt. Same on every turn, with no timestamps, user names or request IDs in it.
3. Stable reference material. Documentation, examples, a codebase summary.
4. Conversation history. Grows, but earlier turns do not change.
5. The newest message. Different every time.

On Claude, the API processes the prompt in the order tools, then system, then messages, and a change at one level invalidates the cache for that level and everything after it. Changing a tool definition invalidates everything. Changing the newest message invalidates nothing before it.

Prompt caching on Claude

On the Claude API you mark where cacheable content ends with a cache_control field. Everything up to and including that block becomes a cache entry. Per the prompt caching docs:

• You can set up to 4 cache breakpoints per request.
• There is also an automatic mode: a single top-level cache_control puts the breakpoint on the last cacheable block and moves it forward as the conversation grows.
• The default cache lifetime is 5 minutes, refreshed for free each time the entry is used. A 1-hour lifetime is available with "ttl": "1h".
• Writing to the cache costs 1.25 times the base input price for the 5-minute lifetime and 2 times for the 1-hour lifetime. Reading from it costs 0.1 times the base price.
• There is a minimum cacheable length, which varies by model (from 1,024 to 4,096 tokens at the time of writing). Shorter prompts are simply not cached, with no error.

Here is an agent turn with two breakpoints: one at the end of the system prompt and one on the latest message, so each turn reuses everything before it.

import anthropic

client = anthropic.Anthropic()
MODEL = "your-claude-model-id"

TOOLS = [...]          # keep this list identical on every turn
SYSTEM_PROMPT = "..."  # no timestamps or per-user text in here

def agent_turn(history: list[dict]) -> anthropic.types.Message:
    # Mark the newest message so the next turn can reuse this whole prefix.
    last = history[-1]
    if isinstance(last["content"], str):
        last["content"] = [{"type": "text", "text": last["content"]}]
    last["content"][-1]["cache_control"] = {"type": "ephemeral"}

    return client.messages.create(
        model=MODEL,
        max_tokens=2048,
        tools=TOOLS,
        system=[
            {
                "type": "text",
                "text": SYSTEM_PROMPT,
                "cache_control": {"type": "ephemeral"},
            }
        ],
        messages=history,
    )

In a real loop, remove the marker from the previous message before adding a new one, so you stay under the 4-breakpoint limit. The docs also note that when a request comes in, the system looks back up to 20 blocks from each breakpoint for an earlier cache entry. In very long conversations, that is a reason to keep a breakpoint near the end rather than only at the top.

Does it pay off?

Treat a normal, uncached prompt as costing 1.0 per use. With the 5-minute cache:

• Two uncached uses cost 2.0.
• One cache write plus one cache read costs 1.25 + 0.1 = 1.35.

You come out ahead the first time the prefix is reused. With the 1-hour cache, the write costs 2.0, so you need two reuses to come out ahead (2.0 + 0.1 + 0.1 = 2.2, against 3.0 uncached). For an agent that takes ten turns on one task, the shared prefix costs 1.25 + 9 × 0.1 = 2.15 instead of 10, a saving of almost 80 percent.

Prompt caching on OpenAI

OpenAI's prompt caching guide says caching is on by default for supported models, so there is nothing to mark. A few things to know:

• Prompts need at least 1,024 tokens to be cached.
• The whole prefix has to match, so the same ordering rules apply.
• You can pass a prompt_cache_key to group related requests, such as all requests for one user or one task.
• Retention options, write pricing and discounts vary by model generation, and newer models add more explicit controls. Check the guide for the model you use rather than relying on a number from a blog post, including this one.

The guide's own advice lines up with everything above: put stable instructions and reference material first, move dynamic content to the end, append to the conversation instead of rewriting earlier turns, and keep tool definitions stable.

Seven ways agents break their own cache

These are the common culprits when the hit rate is lower than it should be:

1. A timestamp in the system prompt. "Today is November 6, 2026, 3:42 PM" changes every minute. Put the date in the latest user message instead, or round it to the day.
2. Per-user text near the top. Move names, preferences and account details below the shared instructions.
3. Changing the tool list between turns. Adding or removing a tool invalidates everything on Claude. Keep the list fixed and restrict tools with tool_choice instead.
4. Non-deterministic serialization. If tool schemas or context are built from a dictionary with unstable key order, the bytes change even when the content does not. Serialize with sorted keys.
5. Rewriting history. Summarizing or trimming earlier turns changes the prefix. When you must compact, do it rarely and in large steps, so you pay for one cache write instead of many.
6. Toggling settings. On Claude, changing tool_choice, extended thinking settings, or adding or removing images invalidates the cached messages.
7. Gaps longer than the lifetime. If a user pauses for ten minutes, a 5-minute entry is gone. For agents that wait on humans, the 1-hour lifetime can be worth its higher write price.

Measure your hit rate

Do not assume caching works. Read it from the response.

On Claude, the usage object reports three numbers: cache_creation_input_tokens (written to the cache), cache_read_input_tokens (served from it) and input_tokens (processed normally, after the last breakpoint). Their sum is your total input. If both cache fields are zero, nothing was cached, often because the prompt was under the minimum length.

On OpenAI, cached tokens appear in the usage details as cached_tokens.

A simple hit rate for Claude:

def cache_hit_rate(usage) -> float:
    read = usage.cache_read_input_tokens or 0
    written = usage.cache_creation_input_tokens or 0
    fresh = usage.input_tokens or 0
    total = read + written + fresh
    return read / total if total else 0.0

Log it per turn. On a healthy multi-turn agent, the first turn shows mostly writes and later turns should be mostly reads. If later turns keep showing large writes, something near the top of your prompt is changing. Diff two consecutive requests and look for the first byte that differs.

Checklist

• Tools first, then system prompt, then history, then the new message.
• No timestamps, IDs or user details in the system prompt.
• Tool list and schemas byte-for-byte identical across turns.
• On Claude: a breakpoint on the system prompt and one on the latest message.
• Hit rate logged on every call.

If you are cutting costs, it is also worth checking how much context your tools return on each call. Large tool responses sit in the history and get billed on every later turn. The post on why agents call the wrong tool covers how to keep them short.

Sources

• Anthropic, Prompt caching (Claude API documentation)
• OpenAI, Prompt caching (API documentation)