# [[o11y-bench]] > An **open benchmark** for evaluating AI agents on real observability and SRE workflows. Built on top of Harbor, runs agents against an actual Grafana stack with synthetic metrics, logs, and traces. Open-sourced by Grafana at [[GrafanaCon 2026]]. ## What it is o11y-bench is a domain-specific benchmark for **agents that do observability work** — root-cause investigations, dashboard editing, query-language fluency. Instead of asking "is this model smart in general?" it asks "can this agent do the things SREs and observability engineers actually do, against a real stack?" It's exactly the shape of evaluation I argued for in [[High INT, Low WIS - What LLM Benchmarks Miss]] — a *skill check with a real DC*, not another general-knowledge ability score. The post calls it out by name as the kind of "specific-and-grounded" benchmark the field actually needs: > *o11y-bench doesn't ask whether a model is smart in general. It asks whether an agent can do the things observability engineers actually do: read production traces, correlate logs with metrics, figure out why the thing that's supposed to be working isn't.* > — [[High INT, Low WIS - What LLM Benchmarks Miss]] ## What it does - Runs agents against a **real Grafana stack** with [[Prometheus]], [[Loki]], and [[Tempo]] — no mocks - Provides synthetic but realistic telemetry data so the same scenario is reproducible across runs and across models - Gives agents access to the [[Grafana MCP]] server so they have the same tools a human SRE would use - Grades agents on **63 tasks** spanning [[PromQL]] queries, [[LogQL]], [[TraceQL]], multi-step incident investigations, and dashboard editing - Built on top of **Harbor** as the underlying agent-evaluation framework - Maintains a **HuggingFace leaderboard** that accepts community contributions - Has a public leaderboard site at [o11ybench.ai](https://o11ybench.ai/) ## Headline metrics o11y-bench prioritises **reliability over one-off luck**, which is why it uses: - **Pass^3** — *consistency across three runs*. The agent has to succeed on all three attempts. This is the metric that distinguishes "actually reliable" from "got lucky once." - **Pass@3** — *best of three*. The agent passes if any of three attempts succeed. This is the more familiar academic metric. The gap between Pass^3 and Pass@3 is itself diagnostic — a big gap means the model is stochastic in ways that would frustrate an on-call engineer. ## Initial results (GrafanaCon 2026 launch) Across **29 model variants** evaluated at launch: - **[[Claude Opus 4.7]]** (reasoning off) — leading on Pass^3 / consistency - **[[Qwen 3.6 Plus]]** — top open-source model, rivalling some smaller frontier models - Frontier closed models still dominate, but the gap is narrower than on general benchmarks - Jack Gordley (on [[Live - Grafana Sigil and AI O11y (Context Window 03)|Context Window 03]]) noted that Qwen "actually rivalled Sonnet and some smaller GPT models" in their internal scoring ## Why it matters (from [[High INT, Low WIS - What LLM Benchmarks Miss]]) The High INT, Low WIS post sets up exactly why o11y-bench is the right shape of benchmark: - **Saturation problem**: General benchmarks like [[MMLU]], [[HumanEval]], even [[SWE-bench]] all eventually get saturated as models cluster at the top. Specific-and-grounded benchmarks saturate too — but more slowly, and they remain more meaningful along the way. - **Skill checks vs. ability checks**: D&D analogy — INT 20 doesn't tell you if your wizard can identify a magical rune. You need the Arcana check. o11y-bench is the Arcana check for SRE agents. - **Co-built with practitioners**: The most useful benchmarks ([[MedQA]], [[LegalBench]], [[FinanceBench]]) are built *with* the people who have to live with the results. o11y-bench fits this pattern — it's built by observability engineers for observability agents. ## Components ### The benchmark harness - **Harbor**, a framework for running agents against full Grafana stacks - Spins up Grafana + Prometheus + Loki + Tempo + synthetic data - Instruments the agent's tool calls, scores against ground truth ### The task suite (63 tasks) - **PromQL** queries — given a question, write the query - **LogQL** queries — log search and aggregation - **TraceQL** queries — trace analysis - **Multi-step investigations** — root-cause work that requires correlating multiple signals - **Dashboard editing** — modify panels, queries, layouts ### The leaderboard - Hosted on HuggingFace - Public submissions accepted - Mirrored at [o11ybench.ai](https://o11ybench.ai/) ## Why I care - It's the productionised version of the argument I made in [[High INT, Low WIS - What LLM Benchmarks Miss|High INT, Low WIS]] — and Grafana shipped it the same week the post went up - It directly feeds into how we evaluate [[Wingman]] (the SRE agent) and [[Grafana Assistant]] - The Pass^3 vs Pass@3 distinction is one of the cleanest framings I've seen of "your benchmark is lying to you about reliability" - Good talk material for the *Asimov's Zeroth Law* / testing-AI thesis ## Related - [[High INT, Low WIS - What LLM Benchmarks Miss]] — my Context Horizon post that calls o11y-bench out - [[AI at Grafana]] — parent - [[Wingman]] / [[Grafana Assistant]] — the agents being benchmarked - [[Agent Observability]] / [[AI Observability]] — Sigil tells you how the agent behaves in prod; o11y-bench tells you how the agent behaves on a benchmark - [[Grafana MCP]] — the toolset agents use to talk to the stack inside the benchmark - [[GCX]] — agents can also drive the stack via GCX - [[GrafanaCon 2026]] — launch venue - [[SWE-bench]], [[MMLU]], [[HumanEval]], [[GAIA]] — the broader benchmark landscape o11y-bench sits in - [[Test-Driven Development]], [[Asimov's Zeroth Law of Robotics - Observability for AI - KubeCON EU 2025]] ## Sources ### Official - [github.com/grafana/o11y-bench](https://github.com/grafana/o11y-bench) — the repo - [o11ybench.ai](https://o11ybench.ai/) — leaderboard site - [Introducing o11y-bench (Grafana blog)](https://grafana.com/blog/o11y-bench-open-benchmark-for-observability-agents/) — the launch post - [GrafanaCon 2026 announcements roundup](https://grafana.com/blog/grafanacon-2026-announcements/) ### Coverage - [Grafana Labs Extends AI Observability To OSS With Open Benchmark Launch — Open Source For You](https://www.opensourceforu.com/2026/04/grafana-labs-extends-ai-observability-to-oss-with-open-benchmark-launch/) - [Grafana Labs Launches AI Observability in Grafana Cloud for LLM Agents — H2S Media](https://www.how2shout.com/news/grafana-ai-observability-llm-agent-monitoring-gcx-o11y-bench.html) ### My own - [[High INT, Low WIS - What LLM Benchmarks Miss]] — calls o11y-bench out as the right shape of benchmark - [[Live - Grafana Sigil and AI O11y (Context Window 03)]] — Jack discusses initial Qwen vs Sonnet results