# [[Observability and AI explained - Grafana Labs]] <iframe width="560" height="315" src="https://www.youtube.com/embed/OTmgMF6gD1k" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> plugins:: "" Related:: "" ## Pre-production ### Thumbnail ### Title 1. agentic observability (31) 2. grafana ai observability stack (46) 3. grafana ai (46) 4. grafana ai observability (46) 5. agentic ai observability (38) 6. ai observability explained (42) 7. what is ai observability (52) 8. ai assisted devops (31) 9. ai sdlc (70) 10. ai in the sdlc (60) 11. ai observability in the sdlc (46) 12. ai x observability across the sdlc (27) 13. ai for observability observability for ai (62) 14. ai and observability (77) 15. ==observability and ai (78)== 16. grafana ai (46) 17. ai agent observability (51) 18. ai agent observability explained (25) 19. ai and observability explained (43) 20. observability and ai explained (42) 21. ai and observability across the sdlc (38) 22. ### Hook AI has changed how we build software. Agents write more of our code, we ship faster than ever -- but every line an agent writes is a line somebody has to operate, debug, and trust. The rate of change has exploded, and change is where trouble starts. Observability was already a hard job when humans wrote everything. Here's the thing, though: the point of AI in observability isn't to replace engineers. It's to hand back the parts of this job that made it exhausting -- so you can understand your systems instead of drowning in them, stay calm when things break instead of carrying the pager weight alone, and spend your energy on the work only humans can do. In this video, I'm going to walk through everything AI-related at Grafana, across the whole software development lifecycle -- and show how AI and observability come together in two directions: AI that observes your systems, and observability for the AI itself. Disclaimer: I'm making this video in July 2026, and all this information is accurate as of that date. I'll leave links in the description below so you can always verify for yourself what has changed if you're watching this video a long time after that. The description also contains the status of each tool I talk about-- whether it's publicly available or only available for preview. ### Structure #### The map: two directions The term "AI Observability" can be vague, because depending on how it's used, it can mean one of two things: - **AI for observability**: AI helps you observe your systems. - **Observability for AI**: you observe the AI itself. These aren't two separate topics — they're woven together across the whole software development lifecycle, so let's go through the different activities and call out what fits where. #### Planning - (Emotion: **Clarity** — *"Can this help me understand what's going on?"*) - In the planning phase, we're trying to decide what feature or fix to build and how to do that. For that, we need clarity on what's going on. - [[Grafana Assistant]] is an AI agent built into Grafana. It knows your telemetry, your dashboards, and your alerts — and it's NOT just a chatbot. It doesn't just answer questions, it does the work: writes your queries, builds your dashboards, and creates deep reports about what it's found, in a new agent-first view called Workspace. - You can ask it questions in natural language. - It supports all the most common data sources so chances are high that you can ask Assistant to analyze your data so you don't have to. - Assistant is not open source, but along with using it on Grafana Cloud, you can also use it with your OSS Grafana instance *- more on that here*. - Assistant can do much more. You'll hear more about Assistant throughout the video, but for the planning phase, Assistant is an excellent partner in helping you figure out how your system is operating, what its weaknesses are, and what you should do next. #### Building - (Emotion: **Momentum** — *"Can I move faster without losing control?"*) - Once you've decided *what* to build, AI can help you build it--faster. But can it do it without you feeling like you've lost control? We have tools that make sure you can observe your code every step of the way. - For example, if you want to instrument your code, you probably want to do so by reusing the same datasources, labels, and metrics that already exist in your stack. - After instrumentation, if you want to debug while you're developing, you can query that same telemetry to see if the new metrics are actually flowing in. You can also use that information to troubleshoot performance issues for endpoints you create. - To help you build stuff, here are some tools you can use. ##### Assistant - Assistant, from within the Grafana UI, lets you add tools to do things like create fixes and raise PRs on GitHub on your behalf, and that can even be triggered from Slack. ##### GCX - [[GCX]] is an open-source structured CLI for Grafana. It doesn't have a model of its own, and instead allows your coding agent (like Claude Code, Cursor, Codex, etc) to interface with Grafana. - GCX works across Grafana products so you can use it to create alerts, query all telemetry signals, create load tests, and more. - GCX also lets you do everything you can do with Assistant on your terminal. You can even link up your Assistant on Grafana Cloud with your local machine so that it can run commands like kubectl in Grafana Cloud for you. Note that the Assistant part is still not open-source-- you just have the option within GCX to link to it as well. ##### Grafana MCP - Sometimes you're using an agent that can't use a CLI like GCX. Maybe you're using a chat app like the Claude Desktop app or ChatGPT. If it allows for MCP connectors, you can use the Grafana MCP to give it access to your Grafana stack. - If GCX is a CLI, [[Grafana MCP]] is like a protocol layer for interacting with Grafana resources. It comes in two flavours: - There's an open-source server that works with self-hosted Grafana and Grafana Cloud, but you run and maintain it yourself. - You can also use Grafana MCP on Grafana Cloud - we host it, and you get to use it without the maintenance burden. ##### Other MCP servers We also have three specialized MCP servers: one for Loki, one for Tempo (now built into Tempo itself), and one for k6. These MCP servers have some skills that are specific to each tool. ##### Summary So which one should you use? - If you're starting, start with Assistant on Grafana and then add some tools to help you build. - If you want to use your own coding agent, start with GCX. Aside from Grafana-specific smarts, it also tends to be more token-efficient than third-party options. - If your agent doesn't have shell access, add Grafana MCP to it. - If you only want to use Tempo or Loki skills, use those MCP servers. - If you want to be able to write load tests with k6, then use the k6 MCP server. ##### If you want to build something else: [[AI SDK]] - If you're building your own agent, we also have an AI SDK: it's an an open-source framework for building agents in [[Go]]. - You can use it to build any agent, but it of course works especially well with Grafana, including built-in instrumentation and integration with Agent o11y, which you'll hear about later. - This is also what we use internally to build some of the tools in this video. #### Operating - (Emotion: **Calm** — *"Can I stop carrying all the operational burden myself?"*) - So now pretend you've got your app or feature built and deployed. Now you're probably going on call for it. Here are some ways you can use AI to prevent the stress of responding to an incident at 3am AND to give you some time back to do more important things. ##### Reactive: Investigations - [[Investigations]] is technically a different product from Assistant, but in practice, Investigations is so fully integrated with Assistant that it seems like it's part of Assistant. - Investigations is an agentic mode for Assistant that helps you triage issues. It can analyze telemetry, produce structured reports with hypotheses for incidents, and - if you're using [[Grafana Incident Response Management]], you can use Investigations to automatically kick off an investigation when an incident is declared. IRM handles the alerting/on-call/incident workflow, and Investigations is the AI analysis layer that can plug into it. - Investigations really helps give you some context on why an incident is happening, and you can even enlist the aid of a swarm of multiple powerful agents to help you get to the bottom of things. - It can also be good for helping you figure out what to do, in case you're on call and handling an issue you haven't dealt with before. Instead of runbooks, we have agent skills to guide you. ##### Proactive - So that's if you're already in the middle of a full-blown incident, but there are also things you can do to stay ahead. ###### Assistant features - Assistant has a feature called Automations that lets you save an Assistant prompt to be run either manually or on a schedule. - You can use this to do things in the background like creating daily summaries, deployment/latency reviews, and other routine repetitive tasks so you catch things before they happen. - Also in Assistant is a [[Watcher Agent]] that you can use alongside static alerts. You can think of this as a sort of dynamic alert-- you describe what you care about, it generates telemetry checks, you review them, and then it watches and lets you know when something looks suspicious. ###### [[Grafana Machine Learning]] - I also want to mention some Machine Learning features here that are part of the Grafana ML plugin. - Don't confuse Investigations with [[Sift Investigations]], which is a ML feature specific to Kubernetes. However, it's no longer being actively maintained. - The ML plugin also has two features that ARE still actively maintained: - [[Grafana Metrics Forecasting]] lets you predict the future behavior of a Prometheus series (or other time-series data source) and optionally alert on it. - [[Grafana Outlier Detection]] identifies data points in a given series that are behaving differently from the rest. - Both are still useful for helping you forecast and spot issues before they happen, helping you breathe easier when you're tasked with maintaining and operating your apps and platform. ###### Adaptive Telemetry - Another use for ML in Grafana is Adaptive Telemetry: it's an ML-driven cost reduction tool across the four main telemetry signals: Metrics, Logs, Traces, and Profiles. It reduces your cost by removing data you don't need while retaining the ones you do. It also reduces the noise you have to sift through when an incident occurs. ##### Summary - So to help you react to incidents quickly: there's Investigations. - To help you prevent them and be more efficient, you can use Assistant, Metrics Forecasting, Outlier Detection, and Adaptive Telemetry. #### Trust / Testing and evaluating - (Emotion: **Trust** — *"Can I rely on this?"*) - An essential part of building something is testing it. [[Doubt is a form of care]], and testing something actually builds your confidence in the shiny thing you've built. So how do we do that for our systems and for AI? ##### Testing your systems *(AI for o11y)* - We have a testing and reliability tool called [[k6 (tool)|k6]] already. It can run browser-level and protocol-level tests, and it now comes with a lot of AI features to help you use it. - On the OSS side - k6 comes with the built-in MCP server that I mentioned earlier - and a command to bootstrap coding agents with k6 skills including the conversion of Playwright scripts to k6 scripts. - On Grafana Cloud - The Assistant has a *k6 script authoring mode* for writing load tests so you can run simple tests quickly or get a huge head start in writing more complex ones. And it comes with best practices like adding thresholds, checks, URL grouping, etc. - Test results can be difficult to interpret, so you can always *use Assistant to explain results* to you and suggest some next actions to improve your app. - Then there's *agentic testing*: an AI-powered, no-code browser testing experience. You describe what you want to do in plain language ("log in, open this product page, click Add to cart") and an agent we host actually executes it in a real browser without generating a script. ##### Evaluating your AI *(o11y for AI)* - So far in this video we've mostly talked about using AI for observability. Now let's talk a bit about observing your AI. What if you're building apps WITH AI and you want to see if they're behaving? How can we trust that AI is doing what we want it to do? - Testing AI is often called "evaluating", but it's really the same discipline with the same purpose of proving you can rely on a system before you trust it. ###### [[Agent Observability]] - Agent o11y is THE way to observe your AI agents. It's an app within Grafana Cloud. - It includes an instrumentation SDK, a database component, and then observation capabilities in Grafana on top of that. - It will walk you through how to instrument your AI agent so that every LLM call is captured as a _generation_ — payload, tokens, cost, timing — and grouped into browsable conversations. Together, this forms a new fifth telemetry signal: agent sessions. - In agent o11y you can have online evals, which are tests that run in production. You can use evaluators as simple as pattern checks to LLM-as-judge to continuously test a sample of your production data. - You can have guards that block seemingly dangerous or sensitive prompts BEFORE they're sent. - It uses OpenTelemetry GenAI semantic conventions and can be correlated along with other OTel-native signals. - It lets you continuously run experiments that test and compare changes against the baseline within your CI pipeline. - You can even use it to instrument your coding agents! ###### [[o11y-bench]] - So Agent o11y helps you evaluate YOUR agents. But how do you evaluate OURS -- or decide which model to trust with observability work in the first place? You benchmark them. - [[o11y-bench]] is our open-source benchmark for AI agents doing observability workflows. It's a set of SRE-style tasks -- querying metrics, investigating issues, building dashboards, managing data sources -- that agents perform against a real Grafana stack, and then get scored on. - It was originally built around the Grafana MCP server as the harness, but it can also run with GCX, so you can compare not just models but the tools you give them. - There's a public leaderboard at [o11ybench.ai](https://o11ybench.ai) backed by a HuggingFace dataset. You can run it against any model -- including local ones -- and submit your own results with a PR. - The point: we don't ask you to take our word for it that AI can do observability work. We publish the test and let anyone run it. #### Collaborating - (Emotion: **Connection** — *"Can this help my team where we already work?"*) - Observability has always been a team sport. Nothing about an incident happens alone: someone notices, someone investigates, someone communicates, someone fixes. The last piece of the puzzle is bringing everything we've talked about to where your team already works -- instead of making everyone come to a dashboard. ##### ChatOps - Assistant is available in Slack, and coming to MS Teams. That means the same agent that knows your telemetry can join the channel where the incident conversation is *already happening*. - Ask it questions about your systems without leaving the thread, kick off an investigation, or have it post the summary of what it found so far. - This also changes the dynamic from private AI chats to shared ones: everyone in the channel sees the same answer, so the context isn't locked in one person's sidebar. - Remember Automations from the Operating section? This is where they land: scheduled reports dropped right into Slack, so the whole team starts the day with the same picture. ##### Assistant on the go - Assistant is also on mobile. Get paged, open the app, and start triaging from wherever you are -- conversations sync across devices, so you can start on your phone and pick up at your desk. - There's also a desktop app that brings Assistant to your local machine: it can work with your local files and tools (think kubectl) alongside your Grafana Cloud data. - And if none of the surfaces we've built fit your workflow, you can call Assistant through its API or via MCP and build your own. ##### [[Assistant Search]] - Part of collaborating is finding what your team has already done. Assistant Search lets you search across your Grafana artifacts -- dashboards, alerts, incidents, past investigations -- by describing what you're looking for, instead of remembering what someone named it three reorgs ago. - It's institutional memory for your observability stack: the answer to "haven't we seen this before?" ### Outro So that's the state of AI and observability at Grafana. We started this video with a distinction: AI for observability, and observability for AI. But if you've been following along, you'll have noticed that the line between them keeps blurring. The agent that investigates your incidents is itself observable. The benchmark that evaluates models uses the same telemetry tools the models are learning to drive. That's not an accident -- it's the whole point. Observability and AI aren't two products bolted together; they're becoming the same discipline. And across the whole software development lifecycle, the pattern is the same. When you're planning, AI gives you clarity about what's really going on in your systems. When you're building, it gives you momentum without losing control. When you're operating, it takes on the burden so you can stay calm. And you don't have to take any of it on faith -- you can test it, evaluate it, and benchmark it until you trust it. Then, and only then, you let it work alongside your team, where you already are. This way you can use AI to improve connection within your team. None of this is about replacing engineers. Everything I've shown you exists to hand back the parts of this job that made it exhausting -- the 3am triage, the repetitive queries, the report nobody wanted to write -- so that we can spend our time on the things that only humans can do: deciding what to build, understanding what matters, and taking care of each other when things break. There's a lot more coming in this space -- keep an eye on the Grafana blog, because things are moving fast. If you want to go deeper on anything I mentioned, I've left links to everything in the description below -- and if you're new to the Grafana stack itself, check out my open source observability video for the non-AI foundations that all of this builds on. ### Related videos and resources ### Pinned comment ``` ``` ### Timestamps 00:00 Intro ## Post-production - [ ] Create captions using Rev, then edit them. - [ ] Take a thumbnail photo and create one for YouTube (1280 x 720). - [ ] Add description. - [ ] Include the title and any keywords in the first few sentences, but in a natural way. - [ ] Add related videos and resources as links. - [ ] Add end screen. - [ ] Add cards if necessary. - [ ] Add chapters/timestamps for YouTube if longer than 5 minutes. - [ ] Add mid-roll ads/check monetization settings. - [ ] Add pinned comment. - [ ] Post video preview on Patreon. - [ ] Schedule on YouTube. - [ ] Add video to relevant playlist(s), or create a playlist if necessary. - [ ] Schedule on Mastodon. - [ ] Create a blog post on [site](https://nicolevanderhoeven.com). - [ ] Add to [video-database](obsidian://open?vault=obsidian-playbook&file=video-database%2FVideo%20database). - [ ] Add to other notes in my vault as appropriate.