# [[Agent Observability]] > Officially called **Grafana Agent Observability**. Internally (and in the SDK, package names, and env vars) it's called **Sigil** — that's the name I sometimes use in my notes. It was also at one point called *Grafana AI Observability*, so that may slip in a bit here and there too! ## What it is Sigil is Grafana's observability platform for teams running [[Large Language Model|LLM]]-powered apps and agents in production. It captures every generation an agent makes, organizes them into conversations, tracks agent versions, and lets you evaluate quality continuously — all as first-class telemetry signals you can view next to your traces, metrics, and logs in [[Grafana Cloud]]. It came out of an internal hackathon led by [[Alexander Sniffin]] (with [[Jack Gordley]] and the assistant team), built on top of learnings from [[Grafana Assistant]] and Investigations. It went from hackathon prototype to a [[GrafanaCon 2026]] public preview launch in under two months. It is **not** the same as the older AI Observability app that was based on [[OpenLIT]] — this is a fresh build. ## What it does - Captures every LLM call ("generation") with full input/output payloads, model + provider, token usage, timing, and metadata - Groups generations into **conversations** by `conversation_id` so you can see full interaction threads - Auto-discovers and catalogs **agents** by `agent_name`, computing versions as a SHA-256 of system prompt + tool definitions (or an SDK-supplied `effective_version`) — prompt/tool changes are versioned with no code release - Runs **online evaluators** over live production traffic and emits scores as metrics you can alert on - Surfaces AI-specific insights: token-budget bloat, system-prompt analysis, prompt drift, dangerous tool calls, [[PII]] leakage - Supports **multi-agent dependency tracking** via `parent_generation_ids` (formerly `depends_on`), building a DAG and propagating quality signals downstream — an upstream eval failure automatically flags all dependents, and the conversation detail view has a graph visualization tab - Enforces policy at the LLM boundary with **guards** — synchronous pre-call checks that allow, deny, warn, or sanitize requests before the model sees them - Routes failing conversations into **collections** via rule actions, turns them into **test suites**, and runs **experiments** to measure whether a change actually improved the agent before you deploy - Integrates with [[OpenTelemetry]] using `gen_ai.*` semantic conventions — works with existing [[Alloy]], [[Tempo]], [[Mimir]] / Prometheus infrastructure - Lets you instrument coding agents (e.g. [[OpenCode]], [[Claude Code]], Cursor via the sigil-sdk plugins) so you can sleep at night while they run autonomously ## Components ### Online evaluation Online evals run automatically over **live production traffic**. You define rules that match generation patterns and attach evaluators that score the result. Scores show up on conversations and generations in the UI, and you can wire Grafana alert rules to them when pass rates drop below a threshold. Four evaluator types ship with the platform: - **LLM-as-judge** — a separate LLM scores responses against criteria you define. Works best with pass/fail rather than 1–10 scales (research shows the scale flips wildly when you reorder the options). Alexander finds this catches the most bugs because you can read the judge's reasoning. - **JSON schema** — deterministic validator for structured output. Good for catching agents that hallucinate field names or eNoom values. - **Regex** — pattern matching. Cheap and fast. Useful for things like "must not mention competitor" or "no API key shapes in the output." - **Heuristic** — rule trees with boolean logic over things like response length, content checks, and emptiness. Faster and cheaper than LLM-as-judge. Pre-made evaluator templates exist for common cases: helpfulness, conciseness, format adherence, groundedness, [[PII]] detection, toxicity. Jack defaults to the PII one for every new agent. Anatomy of a **rule** (the rule decides *which* traffic gets evaluated; the evaluator decides *how* to score it): - **Selector** — which generation type to evaluate (e.g. `user_visible_turn` for user-facing assistant responses only) - **Match filters** (optional) — restrict to a specific agent, environment, or model - **Sampling rate** — e.g. evaluate 10% of matching generations - **Linked evaluator** — which evaluator scores the selection Bool-type evaluators only record a pass/fail verdict if you explicitly configure a `pass_value` on the output key — otherwise you get a score but no verdict, and alerting won't work. Alert rules fire on the pass rate (e.g. "alert when fewer than 90% pass"). ### Guards Guards are **synchronous policy checks that run *before* the LLM call** and decide whether the request proceeds, is blocked, or is sanitized. Where online evaluation is retrospective measurement, guards are real-time enforcement — blocked requests never reach the model. Three kinds: - **Transform guards** — regex-redact PII (SSNs, emails, phone numbers) inline before input crosses the model boundary - **Tool-filter guards** — block specific tool calls by name, with glob support (e.g. `shell_exec`, `file_delete`) - **Evaluator guards** — run an LLM judge or regex at the boundary; `deny` blocks the request, `warn` allows but logs Guards **fail open by default** (the SDK proceeds if the guard service is unreachable), configurable to fail-closed. PII protection is two-phase: (1) GitLeaks-style masking in the SDK before data ever leaves the local environment (layer 1: SSNs, secrets by default; layer 2: optional phone numbers, IPs), and (2) a PII gate guard at the request boundary that can block synchronously and trigger automated key rotation on detection. ### Offline evaluation: collections, test suites, and experiments Offline evals run a **dataset of conversations** through your agent or prompt to make sure it performs against a known benchmark. This is the "test before you ship" side of the equation, complementary to the live online evals. At GrafanaCon 2026 launch the platform was mostly focused on online evaluation, but as of GA the offline side is built out into a full improvement loop (per Jack's AI Week narrative, "Agent O11y as the Full Agent Lifecycle"): 1. **Rules + actions → collections**: when a rule's evaluator flags a failure pattern, an action routes matching conversations into a **collection** 2. **Annotation**: collections are annotated by a human or by an agent via [[GCX]] 3. **Test suites**: annotated conversations become test cases (seed prompt + expected output). Continuously improving the test suite is the highest-leverage part of the loop — hill climbing only works against strong data 4. **Experiments**: the SDK pulls the test suite, invokes the agent, records scores, and assembles a report — token breakdown, run cost, pass rate. Test cases have stable IDs so you can compare two or more runs (e.g. baseline vs. prompt change) to decide if a change is safe to deploy 5. **Deploy + monitor**: the new agent version shows up in the Agents tab with a diff, so you can roll back if a production eval score drops See also [[o11y-bench]] for the related open benchmark for observability agents. [[Promptfoo]] and [[LangSmith]] are the prior art in offline eval. ### Agents / system prompt analysis Under the **Agents** tab in the plugin: - Agent catalog with all active agents, their version history, and tool/prompt footprints - Versions are computed automatically from the system prompt + tool definitions, so any change creates a new version - **System prompt analysis**: grounded in real conversations (not just static linting of the prompt), it flags failure modes, rates tool quality, surfaces token-budget waste, and gives high-priority findings to fix - **Tokenizer view** lets you see how the prompt actually tokenizes for a given model - This is what caught Jack's 20,000-token system prompt that was producing ~10 output tokens because of an unused MCP tool bloating his context window ### Data flow / OpenTelemetry integration Two channels in: 1. **Generation export** — structured generation data (full payload, agent metadata, depends_on) sent over HTTP/gRPC to the AI Observability API. This is the bit that powers evaluators and conversation analysis. 2. **OTLP telemetry** — standard OTEL traces and metrics with `gen_ai.*` semantic conventions, sent through Alloy → Tempo / Prometheus. Recent generation data is stored in MySQL for fast queries; older data is compacted to object storage (S3, GCS, Azure Blob, MinIO). ### The Sigil SDK - Public SDK repo: [github.com/grafana/sigil-sdk](https://github.com/grafana/sigil-sdk) - Internal Grafana repo: [github.com/grafana/sigil](https://github.com/grafana/sigil) (private) - Python package: `sigil-sdk` - Env var prefix: `SIGIL_*` - Languages: **Python (`sigil-sdk`, 3.9+), TypeScript (`@grafana/sigil-sdk-js`, Node 22+), Go (`github.com/grafana/sigil-sdk/go`, 1.23+), Java (`com.grafana.sigil:sigil-sdk`, 17+), .NET (`Grafana.Sigil`, .NET 8+)** - Provider helpers for OpenAI, Anthropic, and Gemini capture generations automatically from LLM client calls - Framework integrations: [[LangChain]], [[LangGraph]], OpenAI Agents, [[LlamaIndex]], Google ADK, [[Vercel AI SDK]] — these attach hooks/callbacks so you don't instrument LLM calls by hand (coverage varies by language) - Coding-agent plugins (Claude Code, Cursor, etc.) live in `grafana/sigil-sdk/plugins`, open source under Apache-2.0 - Built-in PII masking via GitLeaks-style patterns before data leaves the SDK Setup gotchas (from the Iron Infusion training): - The SDK does **not** create OTel providers automatically — without an explicitly configured TracerProvider/MeterProvider, generations flow but traces and metrics silently vanish into a no-op - Alloy / OTel Collectors are **not** generation-ingest receivers — generation data must go directly from the SDK to the Sigil endpoint (an Alloy-based path is coming: grafana/alloy PR #6352) ## Plugin UI The plugin queries three backends: Prometheus (metrics), Tempo (traces), and the AI Observability API (generations). - **Analytics** — activity, latency, errors, tokens, cost, cache behavior. Includes natural-language insights ("token consumption dropped 90% — verify if workload shift"). - **Conversations** — browse, filter, search, drill into individual generations. - **Tools** — tool usage analytics. - **Agents** — agent catalog + version history + system prompt analysis + tokenizer. - **Evaluation** — configure rules, evaluators, and view scores. - **Setup** — onboarding wizard and demo data seeding. - Registered in the Grafana command palette (`Cmd+K` / `Ctrl+K`). ### RBAC Five plugin roles: - **Sigil Viewer** — landing page and tutorial only - **Sigil Data Reader** — dashboards, traces, model cards, evaluation pages, but **no conversation access** - **Sigil Reader** — full dashboard *and* conversation access (not the same as Data Reader!) - **Sigil Feedback Writer** — can write conversation feedback (ratings, annotations) - **Sigil Admin** — everything, including evaluation config, alert rules, and settings ## Why I care - It's the productionised version of the testing-AI thesis I've been pushing in *[[Asimov's Zeroth Law of Robotics - Observability for AI - KubeCON EU 2025|Asimov's Zeroth Law of Robotics]]* - I instrumented my D&D demo app with it and it found bugs my unit tests couldn't (e.g. when the dungeon master narrated a die roll instead of rolling it) - The "instrument your local coding agent" use case is wild — Jack runs evaluators that alert him when [[OpenCode]] tries to delete a file outside the working directory or leak an API key - It pairs naturally with [[GCX]] (the Grafana Cloud CLI) for closing the feedback loop: agent emits telemetry → eval fires → CLI surfaces the alert → agent fixes itself ## Related - [[AI Observability]] — my parent note - [[Live - Grafana Sigil and AI O11y (Context Window 03)]] — Context Window episode 3 with Alexander, Jack, and Tiffany - [[Grafana Assistant]] — the assistant team built Sigil to instrument themselves first - [[OpenTelemetry]] — Sigil rides on `gen_ai.*` semconv - [[OpenLIT]] — what the *previous* Grafana AI O11y app was based on; Sigil is unrelated - [[o11y-bench]] — Grafana's open benchmark for observability agents - [[GCX]] — Grafana Cloud CLI; can drive AI O11y from your terminal - [[OpenCode]] — open-source coding agent Jack instruments with Sigil - [[Alexander Sniffin]], [[Jack Gordley]], [[Tiffany Jernigan]] ## Sources ### Official Grafana - [Introduction to Grafana AI Observability](https://grafana.com/docs/grafana-cloud/machine-learning/ai-observability/introduction/) — the canonical concept doc - [Get started with AI Observability on Grafana Cloud](https://grafana.com/docs/grafana-cloud/machine-learning/ai-observability/get-started/grafana-cloud/) - [Online evaluation reference](https://grafana.com/docs/grafana-cloud/machine-learning/ai-observability/introduction/#online-evaluation) - [OpenTelemetry integration](https://grafana.com/docs/grafana-cloud/machine-learning/ai-observability/introduction/#opentelemetry-integration) - [Press release: Grafana Labs Targets the "AI Blind Spot" — GrafanaCon 2026](https://grafana.com/press/2026/04/21/grafana-labs-targets-the-ai-blind-spot-with-new-observability-tools-announced-at-grafanacon-2026/) - [Blog: 2026 observability trends and predictions](https://grafana.com/blog/2026-observability-trends-predictions-from-grafana-labs-unified-intelligent-and-open/) - [GrafanaCon 2026 announcements hub](https://gra.fan/gcon26) ### Internal (Grafana) - [Iron Infusion Session 4.2 – AI Observability (Sigil)](https://docs.google.com/presentation/d/1YgmrFsb8WWMX32zTNIvgRAHeShhQOtjM9oIUQS7tieg) — technical training deck: architecture, guards, RBAC, SDK setup gotchas - [AI Week Narrative – Sigil as the Full Agent Lifecycle](https://docs.google.com/document/d/16OExAfNTYWAKFtQi01Oh7oQ6gAjWvg3Lm_oGMWheQC0) — Jack Gordley's GA-era writeup of the collections → test suites → experiments loop - Slack: #ai-observability-sigil, #sigil-dev; rename-to-Agent-Observability confirmed in #product-marketing-ai (Mat Ryer's proposal) and announced 2026-07 — as of GA it's **Grafana Agent Observability** ### Repos - [github.com/grafana/sigil-sdk](https://github.com/grafana/sigil-sdk) — public SDK - [github.com/grafana/sigil](https://github.com/grafana/sigil) — internal (Grafanistas only) ### Press / coverage - [SiliconANGLE: Grafana is trying to close the AI observability gap](https://siliconangle.com/2026/04/21/grafana-trying-close-ai-observability-gap-enterprise-agents-reign-supreme/) - [AITech365: Grafana Labs Tackles AI Blind Spot at GrafanaCon 2026](https://aitech365.com/manufacturing/grafana-labs-tackles-the-ai-blind-spot-with-advanced-observability-innovations-at-grafanacon-2026/) - [The Register: Grafana — Free AI for all, please don't bankrupt us](https://www.theregister.com/2026/04/22/grafana_goes_free_with_ai/)