# [[Live - Grafana Campfire May - GrafanaCON roundup]] <iframe width="560" height="315" src="https://www.youtube.com/embed/6qYCSBT3yU4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> Related:: "" [talking points Google doc from live community calls](https://docs.google.com/document/d/15XudGzmAESMAbbz-cUZPeVN6Obkxiou1frtnS1FF9HM/edit?tab=t.0#heading=h.h44khdthcjk) from [[Jay Clifford]] ## Topics ### What is GrafanaCON? Brief history: - Grafana was written in Barcelona in Dec 2013 - First GrafanaCON had 30 people in Squarespace's office - Now: 10th GrafanaCON: 35M users, 7,000+ customers, $400M+ ARR, 1600 Grafanistas across 40 countries Announcements blog: https://grafana.com/blog/grafanacon-2026-announcements/ ### Grafana v13 - New data sources: now 170 data sources x 120 panels - New visualizations: Graphviz panel - Annotations - Dynamic dashboards are now GA (Champion Roxana Turc asked about this) - New: dashboard templates, with curated templates to get started quickly - New: saved queries - library of reusable queries - New: Interactive learning - New: Git Sync - Soon: Grafana Marketplace - Also: Google now uses Grafana #### Grafana Alerting - Now covers everything Prometheus + Mimir + Loki Alertmanager did - On Mimir Alertmanager: it is available and will be supported, though we will be pushing users to move to GMA. We expect to find exceptions in which the customer asks to keep using DMA. The new Grafana Alerting engine covers everything Mimir AM did, and there's a one-click import wizard with a Prometheus compatibility layer — so the migration story is solid. The direction is clearly toward consolidation. ### AI - Assistant everywhere: Grafana Cloud, now OSS + self-hosted, MCP, GCX - GCX: CLI tool - Public preview: AI Observability - Assistant Investigations - ChatOps: Integrations with Slack and Microsoft Teams - o11y-bench ### Databases Common project they converged on: - **Goal:** Reduce TCO, improve availability - Decouple read and write paths through ingesters - Improve inefficient replication - Reduce prohibitive multi-AZ costs - **Tradeoffs:** higher write and query latency - Query engine also had to be rebuilt to make the new architecture really pay off (Mimir QE, Loki QE, Tempo's streaming QE and per-attribute Parquet dictionaries, Pyroscope's stateless query backends) #### Loki - DataObjects: columnar log storage per field (20x less data scanned, 10x faster queries on aggregations) - Object-storage-native index for needle-in-haystack queries thanks to acquisition of Logline (Jason Nochlin) #### Pyroscope - Killed ingesters entirely. Stateless segment writers flush to object storage every 500ms. - 74% infra cost reduction in some cells, 19% gains in other dimensions - New: recording rules from profiles (export profile data as metrics) - New: profile exemplars (drill into a single profile from an aggregate flame graph) - New: span CPU heatmap panel #### Tempo - Same Kafka-based architecture as Mimir 3.0 - Plus new Parquet block format with per-attribute dictionary control (the storage-side fix for the OOM problem) - **"Tiny trace" war story:** queriers were OOMing on a 500-span trace. Spans had trickled in over 7 days with huge high-cardinality JSON attrs attached — creating massive Parquet dictionaries across every block. Reading one small trace meant unpacking gigabytes of dictionaries in memory. - Fix: per-attribute dictionary control leading to 95% memory reduction on affected lookups, letting them safely raise the attribute length limit - Cross-pillar wiring: bidirectional Pyroscope and Tempo linking (click a slow profile > open the matching Tempo trace) and span CPU heatmaps - Grafana Cloud processes 12B spans/24h through Alloy > Tempo internally #### Mimir - Mimir 3.0 - Near-term target: 10 billion active series per Mimir cluster - **"Query of death" war story (March 2023):** Customer's 40 KB PromQL regex maxed out ingester CPUs and took down the largest Mimir cluster. - Regex unrolling: 97% of Grafana Cloud regexes now skip the engine - Cost-based query planner - Ingester overload protection - Multidimensional query scheduler ### k6 - AI-first release. New subcommands: `agent`, `mcp`, `docs`, `explore` - You can configure an agent with k6 skills and an embedded MCP server - Assertions API inspired by Playwright's API - Now it has a formal extension catalog - k6 operator hit 1.0 for distributed testing ### OpenTelemetry - "Be as boring as possible" in the push for CNCF graduation - Vision: `apt-get install opentelemetry` - Alloy is now an official OpenTelemetry Collector distribution (must understand OTel YAML config directly, and it does) - Two engines in one binary, side by side (default alloy + OTel engine that uses native OTel YAML) + engine extension lets both run in the same Alloy instance: currently experimental, on its way to GA - Private preview: Fleet Management with OpAMP: remotely push config to any OTel collector (not just Alloy) from one control plane - Beyla is now OBI, and it was donated to OpenTelemetry. It's now OpenTelemetry's official eBPF auto-instrumentation ### Other talks #### TamaGROTchi ("mean time to feed") - Tamagotchi powered by Grot, ESP32, fully OTel-instrumented to Grafana Cloud (Hackathon winner) - **Real lesson:** how do you instrument resource-constrained devices that can't run a full OTel SDK? Answer: serialise a raw JSON OpenTelemetry packet, POST it over HTTP. Applies to warehouse robots, EPOS terminals, anything tiny. - Wired to Grafana Knowledge Graph with custom SLOs including "mean time to feed," RED metrics, and an Assistant integration that gets noticeably upset when you neglect it. ### Themes - **Boring is the goal** (stability, availability) - **AI all the things** - **Honesty about limits** - Mimir/Tempo talk: 3 production incidents - Supply-chain attack talk: postmortem of a huge security incident - Scaling Assistant talk: consistency issues ### Grafana Champion Questions #### From Vaniash Vaghela - The upcoming changes to the Orgs/teams and how does that affect the provisioning/permissions. - How does the dashboards delete/restore functionality work with the GitSync feature? Does both actions result in a PR or commit to Main? - Will Mimir Alertmanager continue to be supported in the light of new Grafana Alerting changes/improvements announced during GrafanaCon? - Lately there have been a lot of changes happening to the UI e.g. new query panel. Do we expect this trend continue? Does an individual user have an ability to switch between the old and new interfaces or is it going to be tied to the feature flag/Grafana instance version only i.e. system wide change? #### Roxana Turc One topic that I think would be nice to touch on is the dynamic dashboards. I think it goes in the new feature bucket from Avanish. ## Timestamps ## References ## Checklist ### Way before the show - [x] Create a thumbnail. ✅ 2026-05-22 - [ ] Schedule the broadcast on Streamyard, and select YouTube, LinkedIn, and Twitch as the destinations. - [ ] Get the Streamyard invite link and put it in the "location" of the calendar invite to the guest. - [ ] Get the YouTube link to the show. - [ ] Schedule announcement on Mastodon. - [ ] Schedule a Mastodon post for when the show is live to say "We're live now!". ### Just before the show - [ ] How do you pronounce your name? - [ ] What are your pronouns? - [ ] We will be using the talking points, but we don't have to be strict about it. - [ ] Does anyone want to share their screen? We can do that now, and I can show you how that works. - [ ] We'll be streaming to YouTube live. - [ ] You'll be able to see comments, but if you have links, I have to paste it into the private chat. - [ ] You can also use the private chat if you need to say something, but you can also just say it out loud. - [ ] Afterwards, we'll say goodbye to the stream, but please stay on past that. - [ ] Just in case I disconnect... stall for a minute and I'll be right back. ### After the show - [ ] Order captions from Rev. - [ ] Add timestamps to the video (at least four). - [ ] Add any links shared to the description of the video. - [ ] Add the video to any relevant playlists. - [ ] Add to [video-database](obsidian://open?vault=obsidian-playbook&file=video-database%2FVideo%20database) or vault as necessary. - [ ] Create a blog post on [site](https://nicolevanderhoeven.com). - [ ] Promote on Discord (NVDH). ## Granola ### Summary ![[plugins/Granola/Grafana Campfire_ GrafanaCON recap]] ### Transcript [[plugins/Granola/Transcripts/Grafana Campfire_ GrafanaCON recap-transcript|Grafana Campfire_ GrafanaCON recap-transcript]] [[Grafana Campfire - GrafanaCON 2026 recap - (Grafana Community Call - May 2026) transcript]]