Evals Are Just QA With a New Name: How to Actually Test Skills and RAG

I have shipped a dozen AI systems. Agents, skills, an entire second brain that answers questions while I sleep. And until recently I had written exactly zero evals for any of them. That is a strange confession from someone who spent fifteen years in test automation, so let me say the uncomfortable part out loud: most people building with AI right now are doing what I did. They change a prompt, eyeball two outputs, and hit deploy. Ask them whether the system got better or worse and the honest answer is no idea.

That is not a small gap. In 2024, an airline’s chatbot told a grieving passenger he could claim a bereavement discount after his trip. The real policy said no such thing. A tribunal made the airline pay. That bug was not caught in code review. It was caught by a judge, in the legal sense, which is the most expensive possible place to catch a bug.

This post is the long-form companion to Episode 11 of The Agentic Engineer. The video is the 20-minute tour. This is the reference: the actual code, the actual folder structure, the actual golden questions, and the actual numbers from the companion repo, which is public now so you can clone it and run every number yourself.

▶ The repo: github.com/sahajamit/agentic-evals-lab

The reframe: evals are QA in an AI jacket

Here is the thing nobody says at the eval-engineer job interviews that are suddenly everywhere. If you have ever written an automated test, you already own most of this vocabulary. “Eval” is a new word for a discipline that predates LLMs by decades.

An eval is three things. You give the system an input. You apply some logic to grade the output. You get a verdict. If you have ever written a test, that is arrange, act, assert with the labels filed off. The eval crowd calls the same three things a task (the input plus what counts as success), a grader (the logic that scores it), and a harness (the plumbing that runs every task, records what happened, and adds up the score).

Line the two worlds up and the translation is almost one-to-one:

If you have never written a test in your life, congratulations, you just learned the whole vocabulary in one table.

The analogy is honest, but it breaks in exactly three places, and every clever piece of machinery in this post exists to paper over those three cracks.

1. Non-determinism. Same input, two runs, two different answers. A single green run means nothing. You won one coin toss.
2. No single correct output. Ask for a summary and there are a hundred good ones. assertEquals is the wrong shape entirely.
3. Fuzzy oracles. For a question like “is this answer good?”, the truth itself is a judgment call. There is no assertion you can write.

That third crack is the one honest gap between QA and evals. In testing, assertEquals(expected, actual) settles it. For a fuzzy output there is no expected to compare against, and that is precisely the hole that LLM-as-judge fills: you use a model to grade the output against a rubric, because no line of deterministic code can. Hold that thought. It comes back to bite us later, because the judge is a model too, and a model needs an eval.

Now let me show you what this looks like on the two things engineers actually build: the skills they write, and the knowledge their agents stand on.

Use case 1: evaluate a skill

Everyone is shipping Agent Skills right now, and almost nobody is testing them. An engineer at Google DeepMind put a number on this: their SkillsBench work indexed roughly 50,000 skills from GitHub, and the fraction with evals attached was close to zero. We are collectively writing a new kind of software and shipping it on vibes.

Here is why that is dangerous in a way that is specific to skills. A skill is a description plus some instructions. Whether the agent picks it, and whether it uses it right, is model behaviour, not code you control. And skills fail silently. When a skill does not fire, you do not get a red error. You get a confident answer produced some other way. The single most common failure is not a bug in the skill’s logic. It is that the model never loaded the skill at all.

So I built a tiny harness that treats a skill exactly like a flaky function under test. It lives in evals/skills/ in the repo. The skill under test is my real nanobanana image-generation skill, the one that makes my YouTube thumbnails.

The design, and why every choice is a QA instinct

The test cases live in a plain cases.json. Each case is a prompt plus a label: should this fire the skill or not?

{
  "id": "h1-youtube-thumbnail",
  "prompt": "Generate a YouTube thumbnail image for my new video titled 'AI EVALS': a robot inspecting a giant scoreboard, bold high-contrast style. Save it under assets/thumbs/.",
  "should_trigger": true,
  "checks": ["nanobanana\\.sh", "-o\\s+\\S*assets/thumbs"]
},
{
  "id": "n4-svg-in-python",
  "prompt": "Write a Python script that draws a simple logo (a circle inside a hexagon) and saves it as an SVG file.",
  "should_trigger": false,
  "checks": []
}

The set is deliberately balanced, and the balance is the whole point: 5 happy-path prompts that should fire the skill, 5 adjacent-but-different prompts that should not (resize an image, write alt text, describe a photo, draw an SVG in code), and 2 vague paraphrases written the sloppy way a real user actually types. The negatives are the half everyone forgets. A skill that fires when it should is only half a working skill. A skill that also fires on “batch-resize these PNGs” is a liability.

The runner itself encodes a stack of testing hygiene that will feel familiar if you have ever fought a flaky suite:

def run_trial(case, trial, with_skill, model, out_dir, skills, base_skills, system_prompt=None):
    ws = Path(tempfile.mkdtemp(prefix=f"skilleval_{case['id']}_t{trial}_"))
    if with_skill:
        for s in skills:
            shutil.copytree(SKILLS_DIR / s, ws / ".claude" / "skills" / s)
    cmd = [
        "claude", "-p", case["prompt"],
        "--output-format", "stream-json", "--verbose",
        "--model", model,
        "--setting-sources", "project",   # isolate: no global skills/CLAUDE.md/memory
        "--strict-mcp-config",            # no MCP servers
        # no execution, no escape hatches:
        "--disallowedTools", "Bash", "Edit", "Write", "NotebookEdit",
        "WebFetch", "WebSearch", "Task", "Monitor", "TaskCreate", "Workflow",
        "--max-turns", "8",
        "--no-session-persistence",
    ]

Read that command block as a list of paranoias, each earned:

• Fresh temp workspace per trial. Agents cheat. Leave anything behind and the next trial reads it. Every case starts from an empty directory, the way a good integration test starts from a clean database.
• --setting-sources project. The skill is copied into the throwaway workspace, and the agent’s global skills, CLAUDE.md, and memory are locked out. I verified this actually holds: in the ablation arm, nanobanana is genuinely absent from the agent’s skill list. The global copy does not leak in.
• Execution denied. Bash, Edit, Write, and even Monitor are disallowed. Nothing runs. No image is ever generated. This is a deliberate choice: I am grading whether the agent decides to use the skill and constructs the right invocation, not whether pixels came out. Trigger and plan, not paint.
• Multiple trials. Two runs per case, because non-determinism means one green run is a coin toss, not a result.
• Ablation. The same happy cases run again with --no-skill, so the skill’s value is a measured delta, not a vibe.

One piece of vocabulary makes the “run it twice” rule click, and it is worth borrowing from the benchmark world: pass@k and pass to the k sound similar and mean opposite things. Pass@k asks whether at least one of k tries succeeds. That is the right bar when a human picks the best result, like generating code and keeping the version that compiles. Pass to the k asks whether every one of k tries succeeds, and that is the bar for anything unattended: a scheduled job, a bot, a skill firing while you are asleep. The gap between them is brutal. On a retail agent benchmark like tau-bench, the best agent solved roughly 60% of tasks on a single try and fell below 25% when you demanded eight-for-eight. Same agent. Seventy-five percent per run sounds fine until you need three-for-three, where you are actually at 42%. A single green checkmark hides exactly that, which is why I never trust one.

The numbers, and the ones that surprised me

Two trials each, 12 cases, model haiku-4.5. Here is what came back (all of this is in the repo’s docs/FACT-LEDGER.md, deterministic enough to re-derive from the saved transcripts):

With the skill installed it fires 86% of the time. Remove it and the same prompts fire it zero percent of the time, which is the point of an ablation. But look at how the no-skill arm fails: not loudly. The agent quietly starts hand-drawing SVG files instead, or asks me for an image tool. It never says “I cannot do this.” It improvises something worse and presents it with a straight face. That is the silent-failure mode, measured.

The negatives were mostly clean. One in ten fired when it should not have, and the one that leaked through is instructive: the “write a Python script that saves an SVG” case, an explicit code task, still pulled the agent into reading the nanobanana SKILL.md in one of two trials. Adjacent-but-different is exactly where a description over-reaches.

But the miss that genuinely surprised me was a vague one. The prompt, verbatim, was:

i need a thumbnail for this video, make it pop

Zero triggers. Not once. The agent asked clarifying questions (“What’s the video about? Any text or title?”) and ended the turn without ever touching the skill. Meanwhile a paraphrase only one notch sharper (“can you whip up some kind of picture for the top of my newsletter?”) fired both times. One notch vaguer and the skill goes dark. You would never find that by eyeballing.

The plot twist: my grader was wrong too

Notice the checks pass-rate up there is only 0.36, well below the 0.86 trigger rate. Selecting the skill is not the same as constructing an invocation that would actually run, and a plan-check catches the gap. But the honest story is worse than that, and it is the most important lesson in this whole section.

My first grader was broken. It matched the check regexes over the entire transcript. The problem: when the Skill tool loads, its RESULT echoes the SKILL.md back into the transcript, and SKILL.md contains example nanobanana.sh -c generate ... invocations. So my checks were passing on the skill’s own documentation. The first pass-rate I got was 0.57, and it was fake. The score looked healthy because the grader was grading the answer key.

I fixed it by restricting grading to assistant-authored content only, tool RESULTS excluded. The number dropped to the real 0.36. And that is the lesson, applied to myself on camera: graders need evals too. The judge, the checker, the harness, every piece of measurement apparatus is itself software that can be silently wrong. If your eval score looks great on the first try, suspect the eval before you celebrate.

The second skill test: picking the wrong sibling

One skill in isolation answers “does it fire when it should.” It cannot test a failure mode that only appears once you have a family of overlapping skills: picking the wrong sibling. So I ran the same harness against my real Atlassian skills, a base atl router plus atl-jira and atl-confluence, all three installed per trial, and added a new metric: cross-trigger confusion, where firing the wrong sub-skill is the bug.

The interesting cases are two deliberate traps. One is a Jira-only action whose prompt is soaked in Confluence vocabulary:

We shipped a workaround for the login outage. Add a comment on Jira ticket PROJ-456 saying the workaround is live and telling readers to see the ‘Deploy Workarounds’ page on our Confluence wiki…

The words “Confluence” and “wiki page” are right there in the prompt. The only actual action is a Jira comment. Its mirror trap is a Confluence page update drowning in ticket keys and sprint vocabulary. These are engineered to make the agent grab the wrong sibling.

Both traps held, 2/2 each. The wrong sibling never fired once, not even on the cases built to induce it. On this model, at this sample size, skill-vs-skill discrimination was perfect. (Honest caveat: only 4 of those 18 trials are the actual traps. One flipped trap trial moves the trap-confusion rate to 0.25. This says “no confusion observed on haiku at two trials each,” not “these skills can never confuse.” The likely reason it held is that the descriptions are cleanly separated, jira to issues and sprints, confluence to pages and wiki, which is itself a design lesson.)

The gap between 0.83 trigger and 0.44 checks tells its own story. The agent kept selecting the right skill and then abbreviating the command: it wrote atl jira create where the real subcommand is create-issue, and atl jira transition instead of transition-issue. Right skill, wrong verb. That is exactly the class of failure a trigger-check misses and a plan-check catches. Selecting the tool is not the same as constructing a call that would run.

This set also forced two more grader fixes, because of course it did. First, firing had to become an exact-name match, since a substring check counted the base atl router as a trigger for atl-jira and made cross-trigger confusion literally unmeasurable. Second, the agent expresses the same plan two ways (it either prints atl jira search ... as text, or calls the Skill tool with args: "search ..."), so the grader now canonicalizes both into one atl <domain> <args> string before matching. Graders need evals too. Again.

The eval outlives the skill

One distinction worth keeping. Some skills are capability skills: scaffolding for something the model cannot yet do on its own. The eval tells you the day the model catches up and you can retire the skill. Other skills are reference skills, durable by nature, and the eval protects them forever against a model update that quietly changes behaviour.

Either way, when you do retire a skill, keep its eval. The test outlives the code. It becomes a canary for whatever replaces it, and for the model itself. That is a regression suite, which is a thing QA has understood since long before any of this had the word “agent” in it.

The honest framing for this whole section: one model, two trials per prompt, on my machine. It is a sketch, not a benchmark. But a sketch that catches a silent trigger failure is worth infinitely more than a vibe that catches nothing.

Use case 2: evaluate a knowledge base (RAG)

The bigger surface is the knowledge your agent stands on. Most people building on AI right now are building on top of retrieval: a RAG pipeline over a pile of documents. And there is one rule that has saved me every single time.

Never grade the whole pipeline at once. Split it in two.

The left half is retrieval: did the right chunk even show up, and how high did it rank? You score this with classic information-retrieval metrics. No LLM call, runs in seconds, free. The right half is generation: given those chunks, was the answer actually grounded and correct? This half needs a judge.

Why split? Attribution. A wrong end-to-end answer is ambiguous. Was the right chunk never retrieved, or retrieved and then ignored? Only splitting tells you which half broke. And in my runs, nine times out of ten, it is retrieval. You cannot answer a question whose evidence never showed up.

The retrieval half: unit-testable maths, zero API keys

The lab ships four retrieval “arms” over a markdown knowledge base for a fictional store called Loomstead (product docs, refund policies, incident reports, requirements, test-run notes, all fictional, nothing sensitive). The four arms, weakest to strongest:

• keyword: naive term overlap, a stand-in for grep, the baseline everyone starts with.
• bm25: classic lexical ranking, the keyword ranker most tutorials teach first.
• vector: local sentence-transformers embeddings plus cosine similarity.
• hybrid: reciprocal-rank fusion of bm25 and vector, then a local cross-encoder re-ranker.

Every one of them runs on your machine. The models (a MiniLM bi-encoder and a MiniLM cross-encoder) download once, about 120 MB, then it is offline and deterministic. Run it twice, get the same numbers. That determinism is a feature, not an accident: in a regression suite, any change in the score is a real change in behaviour, never noise.

The metrics themselves are the point of the whole exercise, and they are just maths. Here is the entire metrics.py, from scratch, so nothing is a black box:

def hit_at_k(ranked, gold, k):
    """1.0 if any gold doc appears in the top k, else 0.0. 'Did it even find it.'"""
    return 1.0 if gold & set(ranked[:k]) else 0.0

def set_recall_at_k(ranked, gold, k):
    """Fraction of the gold docs that appear in the top k. For multi-evidence questions."""
    if not gold:
        return 0.0
    return len(gold & set(ranked[:k])) / len(gold)

def mrr(ranked, gold):
    """Reciprocal rank of the FIRST gold doc. Rewards ranking the right answer high,
    not just including it somewhere. This is the metric dilution hits first."""
    for rank, doc in enumerate(ranked, start=1):
        if doc in gold:
            return 1.0 / rank
    return 0.0

hit@k is the headline “did it find the doc.” recall@k matters when a question needs evidence from more than one document. MRR (mean reciprocal rank) rewards putting the right doc high, not just somewhere in the list. NDCG is in there for completeness. There is nothing AI-specific about any of this. Retrieval is unit-testable with information-retrieval maths that is older than most of the people writing RAG pipelines.

One command runs the frozen question set through all four arms and prints a scoreboard. On the clean 28-document corpus:

Hybrid finds the right document 98% of the time. No API key, no LLM call, no waiting. It ran in seconds and it was free. That is an eval: 50 tests for a system that cannot be trusted to answer the same way twice.

The golden dataset: 50 frozen questions, tiered on purpose

Everything above is scored against evals/golden.yaml, a frozen set of 50 questions I wrote in advance and never touch during the experiment. Only the corpus grows. Fifty questions, 42 answerable (20 easy, 12 hard-paraphrase, 10 multi-hop) plus 8 deliberately unanswerable ones. The tiers each probe a different part of the pipeline. A few real ones, verbatim:

- id: e02          # easy: single-hop, keyword-friendly
  tier: easy
  question: "What does the FREESHIP coupon do?"
  expected_docs: ["features/coupons.md"]

- id: h02          # hard: same fact, almost no lexical overlap (probes vector search)
  tier: hard
  question: "How many days do I have to send something back if I'm in the States?"
  expected_docs: ["policies/refund-policy.md"]

- id: m07          # multi: synthesis across two docs, scored as set-recall
  tier: multi
  question: "The coupon-stacking incident and its recurrence-prevention invariant
             relate to which two feature docs?"
  expected_docs: ["features/coupons.md", "features/cart-flow.md"]

- id: u01          # unanswerable: the right behaviour is to abstain, not invent
  tier: unanswerable
  question: "What is Loomstead's cryptocurrency refund policy?"
  expected_docs: []

Look at what each tier is doing. The easy ones are keyword-friendly and probe whether basic lexical retrieval even works. The hard paraphrase (h02) deliberately shares almost no words with its source document (“send something back,” “in the States”) so that only semantic search can find it, which is exactly how you catch a pipeline that leans too hard on keywords. The multi-hop ones need two documents and are scored on set-recall. And the unanswerable trap (u01) has no gold document at all: Loomstead has no crypto refund policy, and the only correct behaviour is to say so rather than invent one. That last tier is your hallucination detector. Every question maps to the document that actually answers it, and that mapping is what turns a demo into a scored eval.

The generation half: LLM-as-judge

The retrieval scoreboard tells you what you found. It says nothing about whether the answer built on those chunks is any good. For that you need the judge, and this is where the metrics stop being maths and start being another model’s opinion.

The two that matter here are groundedness (also called faithfulness): how much of the answer actually had backing in the retrieved documents, and context-recall: did the retrieved context contain what was needed. An answer can be fluent, confident, and quietly making things up. Faithfulness is the metric that catches that.

And because the judge is a model, it inherits every model failure mode. Binary verdicts beat a 1-to-10 scale, because nobody, not even the model, can reliably tell a 3 from a 4. Judges have documented biases: position bias (they favour whichever answer they saw first, so run both orders), verbosity bias (padding an answer with filler fools weaker judges over 90% of the time in the research), and self-enhancement (models prefer their own writing). The mitigations are one dimension per judge, always give it an “I don’t know” exit, and above all, calibrate: never trust an uncalibrated judge. The repo’s second-brain harness refuses to even start until the judge has passed a calibration gate, which I will show in a moment.

The dilution experiment: bigger knowledge base, quietly worse

Here is the failure mode nobody warned me about. A knowledge base is not set-and-forget. It grows. And every document you add is a new chance to retrieve the wrong thing. Add a second region’s refund policy and the first one now has a near-identical decoy sitting next to it. Add ten adjacent product pages and the one you wanted hides under nine lookalikes.

Bigger knowledge base, quietly worse retrieval, and no error anywhere. The answers still sound confident. That is the scary part, and it is the counterintuitive core of the whole episode.

So I built three versions of the Loomstead knowledge base and ran the same frozen 42 answerable questions against each:

Only the corpus changes between stages. The questions, the models, and k are all pinned, so any movement in the numbers is caused by the documents I added, nothing else. Full disclosure: this is a stress test. I compressed two years of organic growth into one commit, and the recipe for every decoy is disclosed in the repo so you can check my work. Here is what growth did to retrieval:

Read the top row of that chart as the headline: every retrieval method got worse as the knowledge base grew. BM25, the keyword ranker most tutorials teach first, lost 12 points of hit rate. That is one in eight answers, gone. Five of my 42 questions broke on it. If you shipped a naive keyword pipeline and your KB tripled in size, you would be silently down a chunk of your answers and you would have no idea.

The reranked hybrid pipeline was the most resilient on finding the right document. It held hit@5 at 0.93. That is the reranker paying its rent, and I finally have proof. But look at MRR: even hybrid slipped, 0.81 down to 0.74. Its ranking got worse even when it still found the doc. And it started handing back the wrong document entirely. My favourite betrayal, real, from the run:

I asked, “What does the FREESHIP coupon do?” The right answer lives in the coupon policy. On the grown corpus, the top result was a bug report: an incident about coupons stacking (INC-2967-coupon-stack-bug.md). Same topic, wrong document. And the answer built on that would have sounded perfectly confident.

There is one more thing in that chart that is the most useful lesson of all. It was v2, the plausible near-misses, that did the damage. v3, the pure junk, barely moved the hybrid numbers at all. A good reranker shrugs off marketing copy and changelogs easily. The document that looks almost right is what kills you. Noise is easy. Near-duplicates are deadly. My eyes would never have caught any of this. A frozen set of questions caught it in seconds.

The payoff: pointing the judge at a real second brain

Everything up to here is a fictional corpus. The last harness in the repo, evals/secondbrain/, points the same discipline at a knowledge base I actually live out of: my real cross-project second brain, about 30 human-curated project docs indexed in qmd. The knowledge base is private, so only the aggregate numbers and hand-sanitized examples are cleared for print, but the pipeline is exactly the one above: retrieve the top-k chunks, answer with a headless agent that sees only those chunks, then judge whether that answer is grounded in them.

The calibration gate runs first, and it is mandatory. Before trusting any verdict, the harness generates one real grounded answer and one deliberately poisoned copy (the same answer with three fabricated claims stapled on: written in COBOL, 400 engineers, won a Turing Award), and requires the judge to PASS the first and FAIL the second. Mine did, and the critique correctly named all three fabrications. Only then does the real run proceed.

The healthy numbers, 12 questions, single run, k=5, haiku for both answer and judge:

Honestly, better than I expected. But the entire point of this post is the one that failed. One question. Retrieval simply missed: the right project document never came back in the top results. No error, nothing flagged. And the model answered anyway, confidently, from the neighbouring documents that did come back. It told me a reconcile job “requires verification.” My own notes said that job was “verified end-to-end.” A flat contradiction, sitting inside my own second brain.

• Answer excerpt (sanitized): “the reconcile job runs on schedule but requires verification”
• Judge verdict: FAIL
• Judge critique (verbatim, sanitized): “Claims the reconcile job ‘requires verification’ but the context states it was ‘verified end-to-end’ - contradicts the source.”

That healthy 83% hit rate would never have surfaced this. Hit rate tells you what you found. Only grading the answer tells you when your system is confidently wrong. That is the difference between a metric and an eval, in one row.

(Two honest caveats the video makes and I will repeat: hit@k here is measured against a single expected doc per question, which undercounts a cross-linked KB. One of the twelve questions scored as a retrieval “miss” but the judge PASSed the answer, because the needed fact lived in a linked decision record that retrieval did return. Treat 0.83 as a floor on “found something usable,” not a ceiling. And the answerer and judge share a model, which is why the poisoned-answer calibration matters: it buys confidence that the judge fails obvious fabrication, not that it catches every subtle one.)

Building the golden dataset, the part everyone skips

If you want this for your own system, step one is not copying my 50 questions, and it is not picking metrics. It is reading traces. The experts are unanimous on this and it is the step everyone skips. The method comes from the researcher Shreya Shankar, and it is really just exploratory testing with a fancier name (open coding, then axial coding): take about a hundred real runs, write a one-line note on the worst thing that went wrong in each, then cluster those notes into a failure taxonomy. You stop when new traces stop producing new categories. In one published case study, three failure categories explained 60% of all problems, and a single date-handling fix took one category from failing two of three runs to passing 95%. You do not find that staring at a blended average.

Once you know your real failure modes, freeze them:

• Freeze every failure into a case. 20 to 50 cases, each a real failure that actually happened, like the day the system invented a refund policy. Each with a pass/fail rule so clear that two engineers would agree.
• Tier them. Easy, hard, adversarial, the way golden.yaml does. Balance things it should do against things it should refuse.
• Hold some back. If you tune your prompts and chunking against the whole set, your system learns the test, not the job. Keep a holdout the way you would keep a test set separate from a training set.
• Code gates, judges advise. Never let an LLM judge block a build. Deterministic checks gate CI; judge metrics inform. In this repo the CI gate is pure retrieval metrics for exactly that reason.

Two kinds of eval live in that dataset. Capability evals start at a low pass rate; they are hills to climb. Regression evals sit near 100%; they are tripwires. When a capability eval passes consistently, it graduates into the regression suite and runs on every commit, forever. And a health check I have borrowed from Hamel Husain, who teaches this craft: a suite that passes 100% is measuring too little. Around 70% means it is doing real work. Husain has another line I keep coming back to, and it reframes the whole judge exercise: building an LLM judge is really a trick to force yourself to finally look at your own data. The judge is not the point. Looking is the point.

The third pillar: agents that act

Skills and knowledge bases are the two things most teams evaluate first, and they are where this repo lives. But the moment your AI stops answering and starts doing, the grader changes shape, and that shift deserves at least a signpost here.

The rule that carries straight over from QA: grade the outcome, not the transcript. When an agent reports “I inserted the row, the order is confirmed,” do not grade the sentence. Go look at the database. The world state is the assertion. A confident narration that changed nothing is just the agentic version of a test that passes without testing anything.

For code-generating agents this gets wonderfully concrete, because execution is the grader. Does the code it wrote compile? Does it pass lint? Does it run? Does it pass the existing suite? I have a small agent at work that writes Playwright tests in our house style, and every prompt tweak is a gamble until those gates tell me whether it paid off. The strongest version of the idea is mutation testing, which has been the honest measure of a test suite for decades: deliberately break the code and confirm a test screams. A test that cannot catch a planted bug is not a test. The same logic sits under benchmarks like SWE-bench, where an agent’s patch only counts if it turns a failing test into a passing one.

Grading the trajectory (did the agent take a sane path, not just stumble onto a lucky endpoint) is the harder frontier, and it is enough material for its own post. I will write that one. For now the takeaway is the one that has held since my Selenium days: a green result you never checked against real state is not a result.

The tools engineers actually reach for

You do not have to build the harness from scratch, though building the 40-line version in evals/core.py once will teach you more than any framework. When I reach for something off the shelf:

• DeepEval if you live in pytest. It is pytest for LLMs, and it fits the muscle memory you already have.
• promptfoo for config-driven matrices in CI, especially if you are a TypeScript shop. (One warning, current as of mid-2026: promptfoo is being acquired by OpenAI, which is winding down its own hosted evals product and pointing migrations at the tool it bought. Keep your eval logic in your own repo and your own CI, so a vendor’s roadmap is never your problem.)
• Inspect from the UK’s AI Security Institute for rigorous, sandboxed agent evals.
• Ragas to borrow the RAG vocabulary (faithfulness, context-recall, and friends); implement the two metrics you actually need yourself.
• Langfuse when you want tracing and evals in one place and want it open-source and self-hosted.
• Braintrust and LangSmith if you will pay for polish.

But the runner is the easy part. The transferable skill, the one that keeps its value across every framework churn, is designing the dataset and the verifier. And that, one more time, is a testing skill you already have.

Clone it and break your own knowledge base

Here is what I believe after all of this. Most AI products do not fail because the model is dumb. They fail because nobody can measure whether a change made things better or worse, so every iteration is a guess, and guesses do not compound. Evals turn vibe-shipping into engineering. The real skill was never prompting. It is knowing whether your prompt made things better or worse. And if you came from testing, as I did, that is not a new skill. It is your oldest instinct, pointed at a system that thinks.

So do not take my numbers on faith. Take the repo.

▶ github.com/sahajamit/agentic-evals-lab

git clone https://github.com/sahajamit/agentic-evals-lab
cd agentic-evals-lab
uv run evals/runner.py      # baseline scoreboard on the clean corpus
uv run evals/dilution.py    # same questions, growing corpus, watch the numbers move

Then point the second-brain harness at your knowledge base (a qmd collection, or any folder of markdown), write a dozen golden questions, and run it. Find out how many of your questions break when the corpus grows. Find out whether your second brain is quietly lying to you.

I shipped a dozen AI systems with zero evals. Not anymore. Your turn.

The video version covers this ground in 20 minutes with animated diagrams; this post is the deep-dive with the real code and numbers. All figures trace to the companion repo’s docs/FACT-LEDGER.md and are reproducible from the committed harness. The dilution chart is the actual output of evals/report.py; the hero and the retrieval-vs-generation diagram were generated with Nano Banana. Subscribe to The Agentic Engineer on YouTube for the next episode.