# [[Testing AI]] ![[Testing AI.svg]] There are a few approaches to testing [[Artificial Intelligence|AI]] and apps that use AI: - [[Benchmark-based evaluation frameworks for AI]] - [[Unit testing AI]] - Human evaluation of AI (end-user feedback) ## Why testing AI is hard AI models are non-deterministic — the same prompt can give a slightly different answer each time — so the pass/fail certainty we're used to in traditional testing doesn't hold. The harder problem is what I'd call the **final answer problem**: a response can *look* completely plausible while being wrong, and a plausible-but-wrong answer is more dangerous than an obviously broken one. This means you can't only check the outcome; you have to care about the process too. If the process was wrong, you can't trust a correct-looking outcome, and if the process was right but the outcome is wrong, something else is broken. (A related trap: when AI generates its own tests, it will happily rewrite those tests to make its changes pass.) ## Evals An **eval** is just a test of whether an AI behaves the way you want. A simple one is asking an assistant "who are you?" and checking that it identifies itself correctly; more complex ones check whether it completes a real task. The reason evals matter is that when you change an AI system — a new prompt, a new tool, a bug fix — evals are how you know *with confidence* whether the change actually made it better. Without that signal, you're guessing. A useful split is **offline vs online evals**, which maps neatly onto pre-production vs production testing: - **Offline evals** run during development, like integration tests in CI. You keep a fixed "golden set" of tasks so that every change is measured against the same bar, and you can drill into specific capabilities. Because it's controlled, you can hammer it and test every variant, prompt, and implementation — not just what's deployed. The catch is you have to anticipate what users will actually do. - **Online evals** score real production conversations (or a sample of them), much like synthetic monitoring or real user monitoring, but with an AI flavour. They're a good pulse check — they can catch quality drops you'd never see offline, e.g. when a model provider silently swaps the underlying model — but they only test the deployed variant and are harder to verify. ## Grading: how you decide if an eval passed Grading (sometimes called using **verifiers** or **graders**) is the genuinely hard part, since the output is natural language. Common approaches: - **Hard-coded checks** — deterministic assertions (length, format, must-cite-X), useful when you control the environment and know the right answer in advance. - **LLM-as-a-judge** — using another model to grade a transcript ("given this task and this answer, was it good?"). Scalable, but you're relying on an LLM to judge an LLM, so it's less robust and harder to fully trust. - **Fact-based rubrics** — the grader independently runs a known-good query against the environment and compares the AI's answer to the real result. This catches subtle failures where the answer *sounds* right but the underlying number is fabricated. Two metrics worth distinguishing when you run a task multiple times: **pass@k** (got it right at least once out of k tries — rough capability) vs **pass^k** (got it right *every* time — reliability). The gap between them tells you whether the model can occasionally do something versus dependably do it, and that gap can be large. ## The evaluation loop If your evals are robust enough to produce a clean signal, you can close the loop: **measure → make a change → measure again**, repeatedly. The key insight is that this loop can be driven by a coding agent rather than a human, because an AI system is "programmed" in natural language and there's no clear, code-like relationship between the instructions you give and what the model does — so the only way to know the impact of a change is to run the evals again. This is what makes consistent improvement possible at a scale no single person could track manually. ## Example: how Grafana tests its [[Grafana Assistant|Assistant]] Grafana is a concrete example of these ideas in practice (not a template for everyone — most of this requires real infrastructure investment). They run tasks through a **harness** (the agent plus its tools) against a real-ish **environment**, then grade the full transcript. To keep results reproducible, they generate synthetic data (so the correct answer is known up front) and use fact-based rubrics that query that environment directly. Their public benchmark, [[o11y-bench]], separates harness / environment / tasks so different agents can be compared fairly, and reports both pass@k and pass^k. Internally they also run a private superset and an automated eval loop. More detail: [[Building an evaluation loop for Grafana Assistant]] and [[Live - TCW 05 - Overview of AI Evaluation]]. %% # Excalidraw Data ## Text Elements ## Drawing ```json { "type": "excalidraw", "version": 2, "source": "https://github.com/zsviczian/obsidian-excalidraw-plugin/releases/tag/2.1.4", "elements": [ { "id": "4y8R7iOA", "type": "text", "x": 118.49495565891266, "y": -333.44393157958984, "width": 3.8599853515625, "height": 24, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "roundness": null, "seed": 967149026, "version": 2, "versionNonce": 939059582, "isDeleted": true, "boundElements": null, "updated": 1713723615080, "link": null, "locked": false, "text": "", "rawText": "", "fontSize": 20, "fontFamily": 4, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "", "lineHeight": 1.2 } ], "appState": { "theme": "dark", "viewBackgroundColor": "#ffffff", "currentItemStrokeColor": "#1e1e1e", "currentItemBackgroundColor": "transparent", "currentItemFillStyle": "solid", "currentItemStrokeWidth": 2, "currentItemStrokeStyle": "solid", "currentItemRoughness": 1, "currentItemOpacity": 100, "currentItemFontFamily": 4, "currentItemFontSize": 20, "currentItemTextAlign": "left", "currentItemStartArrowhead": null, "currentItemEndArrowhead": "arrow", "scrollX": 583.2388916015625, "scrollY": 573.6323852539062, "zoom": { "value": 1 }, "currentItemRoundness": "round", "gridSize": null, "gridColor": { "Bold": "#C9C9C9FF", "Regular": "#EDEDEDFF" }, "currentStrokeOptions": null, "previousGridSize": null, "frameRendering": { "enabled": true, "clip": true, "name": true, "outline": true } }, "files": {} } ``` %%