# [[Interview with Will - Investigations Arena]] <iframe width="560" height="315" src="https://www.youtube.com/embed/YPfqchw8LCo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> plugins:: "" Related:: "[[Investigations Arena]], [[Testing AI]], [[Grafana Assistant]], [[Grafana Assistant Investigations]]" ## Transcript Speaker 1 (00:00): Hey, my name is Will. I'm a software engineer at Graphana. I've been there for three years and since one year I've been focusing on AI. So the Graphana assistant, investigations and memory features. Speaker 2 (00:14): And what's the latest thing you've been working on? Speaker 1 (00:17): So we developed investigations V1 and we saw that we hit the sailing at some point and we came up with a different approach, but this time we needed good evaluation and demo environment or this was not enough because this cases were too easy and too far from the reality of production environment. So we decided to build an arena. So it's a little bit like the idea of bot programming fighting against each others for specific tasks or games, for example. In this case, the topic was investigations in our production cluster, real incidents. And we started building these bots that had different strategies, different tools, these were AI agents. And it allowed us by comparing a diversity of strategy to surface the best approaches and from there to iterate and bring more test cases through more incidents and reach higher and higher quality. So the idea here is that the bots will try to solve the same case and we use an LLM as a judge to compare the different approaches and we have different criterium because for the incidents we have a verified root cause by a human. (02:02): So this gives us our ground roof to compare and the LLM as a judge will compare this ground truth against the answers that the agents came up with. (02:18): And the important part at first was not even the score, but these were the insights. This tool was better with this strategy and this ones has maybe some noise in the results that poisons the context or the returned result from this specific tool are like it's too much. So we managed to really optimize bit by bit and come up with the investigation V2 that we just released. And it's a massive improvement compared to our V1 approach. Speaker 2 (03:01): So is it comparing just different models for investigations or is it also comparing GCX and NCB versus assisted? Speaker 1 (03:12): Right. So both actually. So we have different versions of our harness for investigations that we run with Sonet and with Opus, but we also use cloud code harness with GCX with Graphena MCP and even just using Kurl to directly hit our public API. Speaker 2 (03:39): Do the models ever cheat? Speaker 1 (03:43): So far it's been quite good. We have guardrails. So basically because we investigate on past incidents, we need to make sure that the LLM only looks at a specific time window. So only when the incident started because ALC would be able to find the pull requests that were used to solve the incident. But they're pretty good at following the rules. They take it as a game a little bit. So far, we have LM that comes, so an agent that checks afterward with the judge to make sure that they followed every rule and they touch it. But actually it's funny because we had the case where using GCX, it started to use some tools that we did not think about in the first place and they were actually very useful for investigation, but we never thought about them and this was like we brought them to our investigation harness and it helped a lot actually. (04:55): So giving the agents freedom and creativity in this kind of environment is very useful. It's like this diversity that sponsor these new ideas that we can put in our work. Speaker 2 (05:12): Cool. Do you have any plans to publish this and share this so that other people can do it too kind of like Oli Bench? Speaker 1 (05:22): Yeah, I think why not? I think we can go this way also. I'm sure this would be useful for other agents. And it would be very nice if we could also compare agents from other people, other companies maybe with different harnesses, maybe even different tools, because right now it's mostly about our MCP, our CLI or our investigation harness. But if the community comes up with their own investigation agent, I think I have seen a few, it would be fun to put them in the arena and help not just our product, but any product to improve in quality.