Nicole (00:00:02): Hi everyone. Welcome back to another Context Window. After a couple months away, AI Week was kind of the big thing that happened there, but we're back and today we're going to be talking about context. Yeah, like six episodes in, we're finally talking about the context window in a live stream called The Context Window, but it's never too late, I think. And we're going to be talking to a couple people who deal with context a lot. And let's get started with Ioanna over here. Welcome to the stream, Ioanna. Hello, Ioanna (00:00:39): Hello. Happy to be here. First time for me, so hopefully it's going to go well. So yeah, my name is Ioanna, Ioanna Armouti. I'm software engineer at Grafana Labs. I joined about three and a half years ago now, and for the last year and a half I'm working in the Grafana Assistant team. And for some months now I'm working heavily on Assistant Search, which is kind of exposing more context to Assistant. So yeah, that's the main job, surfacing more context to Assistant. Yeah. Nicole (00:01:11): Great. And over down there is Ben, who you may have already seen because we did an episode with him on MCP, CLI and gcx. It was like a jam-packed episode. So I was like, better get Ben on when we actually have time to listen to him talk because it was five people on there, all of us super talkative. Ben, would you introduce yourself? Ben (00:01:35): Yeah, absolutely. Yeah. So there are a lot of us that went there. Yes, I'm Ben Sully. I'm also in the AI department working on things like the Grafana MCP server and gcx and various different Assistant things. So a lot of my job involves figuring out, like you want to, which things belong in the context window and how can we create tools that return sensible context and don't overwhelm it. I've heard the word context more in the last year than I think all the other years in my life combined. Nicole (00:02:16): Yeah. Well, before we fully get started, since I mentioned AI Week, there is a video that we released because that was like, if you don't know, there was one week in the end of July when we released something, at least one thing, but actually multiple things a lot of the days, every day of the week. And it was kind of full on. So I made a video if you want to catch up on the state of everything AI at Grafana. But yeah, let's get started. And I wanted to start with this thing that Claude released in the blog a few days ago actually. And in this blog, they said that they went through a lot of the data from the last six months of how developers use Claude Code. And one of the findings that they had was that the context per request has grown 2.6 times. (00:03:16): So the input to output token ratio used to be 189 tokens of input that is being read for one token of output, which is already, when you think about it in those terms, I know a token isn't equal to a word, it's like three-fourths of an English word. But if you just think about it as words, it's like 189 words for every word of output. That's kind of Ben (00:03:42): A lot. Yeah, that ratio is huge, right? It's massive, Ioanna (00:03:44): Yes. Ben (00:03:45): I would never - Nicole (00:03:46): Now it's 324. Ben (00:03:49): Yeah, Ioanna (00:03:50): It exploded. Ben (00:03:52): Yeah. Nicole (00:03:52): It's just like - Ben (00:03:54): That's ludicrous, isn't it? And do you remember not so long ago that the context window was a maximum of 200,000 for all these models and you were like, oh, I cram everything in there. And then suddenly everything has a million tokens now. Nicole (00:04:08): Yeah, now a million's like whatever, everyone has that now. Ioanna (00:04:11): Yeah, normal. It has evolved so much and I think this is how it's a proof that context is really valuable and really powerful in an agent. So yeah, putting the right stuff inside this context, I think it's what we should work on. We're putting a lot of effort into it. But yeah, it has exploded. I do remember agents just cutting the conversation, ending the conversation because of context and now it's not the case anymore. Nicole (00:04:39): Yeah. And we're just expecting a lot more of our agents too. In the same article, they say that developers are twice as likely to connect a tool server or use a skill. That really speaks to the maturity of the industry and the people that are using Claude Code maybe specifically. We don't just expect someone relatively smart. We expect an expert. We expect someone who knows our stack or has a lot of information about us, which is also kind of scary. Ben (00:05:13): Yeah, agreed. And it sort of points to the direction that we're going where things are more and more automated and we expect agents to take on these tasks with smaller prompts and with fewer and fewer explicit instructions in the prompts that we write because you give them tools to solve these problems themselves rather than having to be told what to do explicitly in the prompt each time. People rely on tools being available for those agents to be able to do those things. It feels like a big shift in the last year for sure in the way people have started using coding agents rather than pasting things into ChatGPT, for example, or into Claude AI and seeing what it spits out. You now expect the agent to go and do these things. Ioanna (00:06:08): It started by an agent just suggesting a paragraph, just a small test or a very small routine or something like that. No one was expecting at the time that it could be able to automatically discover your telemetry from Grafana and dig into your logs and discover bugs and issues inside your code. And now this is kind of a given, you're expecting your coding agents to have access to all this data and use it effectively and efficiently. Nicole (00:06:37): Yeah. Well, I think maybe it would be useful to talk about what context is in the first place. And I like to think of it as chatbots aren't the only application of AI, but it is one that we were familiar with before AI because there would also be chatbots and before that there were human customer service representatives. And I kind of think of context in that sense when you talk to support by chat or something. Context would be things like my account number or maybe some authentication that I am myself, those authentication questions, maybe in some cases my address or a customer ID or something like that or how much was my last utility bill or whatever it is. All of that would be context that I would want the agent, the customer support agent to know. So in terms of AI, how would you define context? (00:07:36): How would you explain it to someone? Ioanna (00:07:40): Yeah, context can be anything for me. Context can be on the observability/coding agents, context I guess, is for me at least it's tools, what tools are available for your agent to use. It's knowledge from multiple places around your stack. It's rules or some skills that you want to have. So if you want to have your agent, you want to have some, I don't know, instructions on what to do some things. So this is context for me at least. Ben (00:08:18): Yeah, I'm with you. I think it's kind of everything, everything that you give to the agent to let it solve the problem and to improve its ability to get to the right solution. And you often need... Yeah, it's a tricky balance and I think we'll come into this into how do you provide enough context, but not too much context. Too much. And not the wrong context because it's easy to point you in the wrong direction by linking to too many things that are irrelevant or that are poisonous, I guess in a sense. Yeah, I think maybe it's becoming a little bit hidden what actually ends up in the context window. Do we ring a little bell when we say the name of the... Yes. Ioanna (00:09:12): I think Ben (00:09:13): We could. I remember I was talking to someone the other day about this and we have so many skills now at Grafana Labs and I think a lot of people probably are in the same boat. And Claude Code is very intelligent. It doesn't eagerly load all of those skills into the context window when you begin. Same for most coding agents. Instead, it loads the name and the title and the description and that is enough for the agent to then be able to load that skill on demand if it wants to. And the idea was that gives the agent just enough context, but not too much to be able to then progressively load that content context in future. But now we have so many skills and so many tools that even just the tool names and descriptions and the skill names and descriptions can add up to tens of thousands of tokens of context.You're already kind of bombarding the agent with hundreds and thousands of things that it has to filter through to be able to solve the task. (00:10:13): So I think one of the things that the LLM providers that Anthropic and OpenAI have been doing is to look at usage patterns and see how this is evolving and make their models better at doing this specific task, digging through available context to know when to load further context. Ioanna (00:10:41): So nothing has to be uploaded first and yeah, you're deferring a lot of this context and then your agent is intelligent enough to go and pick up whatever it needs. So yeah, Ben (00:10:55): That's been a huge Ioanna (00:10:55): One. Ben (00:10:56): We're leaning quite heavily on the harnesses there as well as just the models. But yeah, that seems to be the way that things are moving at the minute is getting better at progressive disclosure. Nicole (00:11:11): Yeah. We've talked about the trade-off a lot and it occurs to me that we have a lot of databases too at Grafana and this is also a database problem. It's like how much do you index upfront versus how much do you value the speed of ingestion and just want to offset that into the read path later? So it's either you can't really have both. Everyone would want for the read and write path to be super fast, but you kind of have a trade off because you can ingest a lot and index everything and then the writing becomes really, really slow. But then later when you're reading it, it's super fast. Or you can do the opposite like what Loki does where it doesn't index everything. So the writing is super fast, but then when you read it later, there is a little bit of a performance hit. (00:12:10): It's not a question of do you want a performance hit? It's going to be there. It's like where is it going to be? It seems kind of similar. Ben (00:12:18): It's a trade off, isn't it? Yeah, it is. It is. And I guess the trade off here is how many tokens do we want to use up to make this context automatically available to the model? And it does feel like we're going with agents getting smarter and smarter, we're moving towards the defer as much as possible and the harnesses are going to be so good at searching those tools using what is effectively the index that you mentioned. So I think that's positive because it saves everyone tokens, hopefully makes these input context windows slightly smaller. Ioanna (00:12:52): Yeah. And also, as we said at the beginning, the actual models have evolved a lot, so the context is increasing. So I think that one of the evals that we have in Grafana, something that we look actively at is how much context we use in advance, when the conversation starts, how much context do we need from the user? And I think that this is quite powerful, but this kind of dynamically changing at the same time because models are evolving. So it's a problem in the sense that you don't want to eat up a lot of context, but at the same time it gives you a bit of room to put more relevant context in there. So again, I think Ben said it already, it's a balance between how much you want to eat from the user. I'm using a very weird phrase here, but how much you want to take out in advance compared to how much of this context is actually useful and can give you a lot of good results afterwards in the conversation. Nicole (00:13:46): Yeah. Even the language that we're using, it's like eating up or spending. I like this article by Anthropic because they talk about context engineering or context window as an attention budget. So it's a finite amount of tokens of attention and you choose what to spend it on. And they also go on about how it used to be that we talked about AI in terms of prompt engineering. It's about how the input that you feed to the AI, but not the amount of it. We were just worried about how to say something. But now it's not just how you say it, it's how much of it you want to say at all. And I like that they have a quote here. They define good context engineering as finding the smallest possible set of high signal tokens that maximize the likelihood of some desired outcome. So it's the smallest possible. (00:14:55): It's like minimum viable product, as small as possible, but still enough. Enough. Do you agree with that definition? Ben (00:15:06): Yeah, I think so. Yeah. And it speaks to the probabilistic nature of how long that they are... It talks about likelihood here, which is a statistical term that you want to maximize the likelihood of getting the desired output that you want by having exactly the right tokens that result in just the correct matrix transformations happening inside the LLM so that they can end up with the correct output, however many tokens down the line that is. And it's often difficult to think about how those things connects under the hood, but at the end of the day, it is about activating the right neurons inside the model so that it ends up with the right output tokens and gets to the right solution if you think about it in a really low level technical way. Ioanna (00:16:02): I think that this is a silver line that we're spending a lot of our time as well into. I've seen a lot of, on this context discovery effort that we're doing in Grafana in the last few months, I've seen a lot of cases where we're giving to the agent context. We're trying to give it the smallest possible as it says there, but there are many times that it's fixating on the information that you're giving to it. So it kind of sees it as a single source of truth, so it doesn't really continue to discover more data around it. And on the other side, you've seen that you're giving too much context to the user on the other hand, sorry, to the agent on the other hand, and then suddenly it spends too much of the resources trying to evaluate all this context that you gave it. (00:16:46): So it's what it says there in this sentence, I think that this is where we put a lot of our effort into. So the silver line there. And it's not trivial. It's definitely not trivial. Nicole (00:17:02): I think it could be kind of confusing because especially, Ben, you're talking about how now models have more than a million tokens in the context window, and if they didn't know any better, somebody could very plausibly look at that and think, "Okay, so I have a million tokens and I need to spend it. Okay, I'll just put everything. Everything that I can think of anyway, it's a million tokens." But it's actually, that's so confusing because even Anthropic says, "No, it's still the smallest amount, so don't use the million if you don't have to." Ben (00:17:42): Right. I guess they call it a budget, don't they? Is it called a budget? But you don't want to maximize it, but it's not like a department budget for department work where if you don't spend it, you lose it next time. No, because as you ask more and more questions and as you go through more and more turns with the agent, every output token ends up being an input token in the next turn. So you need to make sure that there is enough left for future turns in case you need to ask further questions, for any tool results, for skills that decide to be loaded or even just the models. I don't know whether the model's thinking counts towards the context budget, but I'd imagine it must do. Ioanna (00:18:26): Should. Ben (00:18:27): Yeah, I'm not sure. Yeah, I would imagine so. Nicole (00:18:31): I know this has happened to me too where I've encountered the lost in the middle problem and I totally freaked out. I was talking to my agent and then there's a little counter for where you are in the context window. So I knew I was well within it. This was still when it was like 200,000. So I was about 100,000 in. It was a long conversation, but still well within it. And suddenly it started talking nonsense. I felt like it was ads and I'm like, "Am I in an episode of Black Mirror? What's happening here?" And it was just talking about random things and I freaked out and I thought, "Have I been hacked somehow? What have I done? What have I clicked on?" But no, it was just sometimes there's this weird phenomenon called lost in the middle problem where even if you haven't exceeded your context window, agents just lose it. Ben (00:19:29): And Nicole (00:19:30): Over and over again, we see studies that say, "Yeah, the more context you use, the worse the results are going to be." So in other words, it's not like within that context window you're good and the second you're out, everything, the quality drops. It's like a slow degradation. Ioanna (00:19:51): It's easy, I think, for a conversation to develop quickly. We've seen it happening, even within Grafana Assistant, we've seen it happening quite a lot. So yeah, it's definitely a problem. Ben (00:20:04): Yeah. And I think we'll talk about some strategies for dealing with that. But yeah, one of the big ones is just making sure your tools don't return hundreds of thousands of tokens. And fortunately the harnesses, Claude Code or Codex or Grafana Assistant have methods for dealing with that now built into them. So even if a tool does go crazy and return an enormous JSON document, you don't end up using too much of your context, fortunately. Nicole (00:20:36): Well, why don't we dig into Assistant Search, Ioanna, because you're talking about Assistant already. I'd love to hear about how this happens in Assistant and maybe we can talk about some strategies you've used along the way. Ioanna (00:20:53): Okay. Yeah. So Assistant Search is a product in a way, it's a context surfacing product that evolved some months ago. I think it was early this year, earlier this year. Early in the year actually. We have September, but yeah, I think it happened January, February, something like that. One of the things that comes to mind when you talk about Grafana is dashboards, visibility. You have a lot of data in there and the data are not very easily searchable from the user. You have a lot of information inside dashboard panels. You have a lot of information inside the alerts. You have descriptions, you have tags, you have nice rows inside the dashboard that are not searchable by the user because search in Grafana especially at the time was mainly lexical search. So you're searching for a dashboard and then the name that you're giving to Grafana needs to match. (00:21:47): But there is a lot of context there that could be useful for an agent and a lot of metrics, a lot of queries that the user cared about to put them in a dashboard to create an alert about this. So these kinds of seem very useful context at the time to surface to the agent through the Assistant. So this is how it kind of started. We started indexing a lot of resources around Grafana. We are using vector storage. So yeah, we're using embeddings and embedding, just to give a very small definition. I'm not sure that they can give a definition of embedding, but it's like a numerical representation of a resource, of a word, of a phrase, of a sentence of something. So now instead of limiting your search in Grafana for some resources on lexical matches, on keyword matches, you're kind of opening it up to semantic to meaning. (00:22:46): You can search by meaning, you can search by concepts. So maybe I'm searching about machine learning dashboards and then suddenly I'm getting dashboards that have machine learning meaning into it in a way, like inside the panel or some metrics inside some alerts. So yeah, this is how it started. And now we are embedding a lot of information, a lot of resources in Grafana, and we're giving this context back to the agent when it does its initial discovery. So when the conversation starts and end user is asking, "Okay, what dashboards do you have that are consuming this metric?" This is not an information that Grafana search can easily surface, but now with Assistant Search, a user can get all the relevant dashboards that they're using this metric, or they can get all the relevant dashboards about the service, alerts, incidents that have happened in the past that are related to a service. (00:23:48): So yeah, I think that I took longer than I should have here. Yeah, this potentially exist. So yeah, the problem is just try and surface a lot of useful information from different Grafana resources to the agent and also to the user. The consumers shouldn't be only the agent. It's a user as well. Nicole (00:24:12): Yeah. I wonder if you can talk a little bit more about embeddings. I know when you download an open weight model and you start it up for the first time, some of them ask you for what context you want to embed and it goes through and there's a process, a period of time where it's processing that somehow. Is it actually indexing that? Is embedding like indexing? Ioanna (00:24:40): So embeddings, it's a numerical representation of something. It represents a semantic meaning of data like words, like sentences, images, whatever you want to index in a way. So what we are doing in Grafana, and this is how it works, at least in my head, is that we're getting a lot of information in advance and we are indexing it. We are putting it inside a vector storage where data, meaning queries become numbers. And then when someone like this data is already indexed in advance and we are dynamically updating this embeddings as well. So when the dashboard is getting updated or when a dashboard is getting created, this data get embedded dynamically. So we create more numbers or we update the numbers in this vector storage. And when a user on an Assistant context, let's say, asks about something, then there is a tool inside a system that will go and search this DB and say, "Okay, what is the related information you have about X, about machine learning, about Grafana system, about, I don't know, Tempo or about the checkout service that they have on my stack?" And it will reveal all the information will go in parallel. (00:26:01): It will do a fan-out query on all the different collections that we embed, like dashboard panels, alert rules, incidents, infrastructure memories. We have our own collections as well, and it will give back to the user all the relevant information. So imagine it as a big place with numbers that are creating groups inside the storage. And then when a query happens, the query again transforms to a number and then we go and find all the numbers from the same group sort of. Yeah, it wasn't the ideal definition, but hopefully it gave a... I don't know if you - Yeah, no, it's good. Ben (00:26:42): I think so. No, not really. I mean, the example I was coming back to was that when I first found out about the idea of embeddings, which was way back when I think word2vec became a thing, and the idea was that you'd give each word in the English language, for example, a vector representation, which is the same as an embedding. And you'd end up with this idea where you have an end dimensional space, maybe a hundred dimensions depending on how large your vectors are. And if you plotted those words out or your words out in this space, things that meant the same would appear in a similar space. And you can even do maths on them to say if you add the words queen and man, you end up with the word king. I think that was the key example that they used there. (00:27:34): And that allows you to do this similarity search by doing some kind of calculation like cosine similarity or something. Nicole (00:27:44): So when I hear this, how much you're processing, I think indexing already takes up so much time and it's a huge undertaking that you have to do at the start, but this sounds even more involved. It sounds like you're still doing the lexical stuff, but also semantic. Yes. It's a semantic search as well. So to me, that seems like this is going to be even more stuff and take even longer. How do you cram all of that into a context window or do you cram it all into a context window? Ioanna (00:28:23): So yeah, we're doing this hybrid search pattern at the moment. We're doing lexical search, keyword searching, and we're also doing semantic, especially in the Assistant context. And this is something that will gradually happen in Grafana search as well. But yeah, this is how we do it at the moment. And we're going into this context engineering concept that we talked at the beginning. Now Ben (00:28:46): We're Ioanna (00:28:47): Surfacing too much context. You're getting 30 relevant dashboards, you're getting 30 relevant panels from dashboard. You're getting 10 alert rules that seem very relevant to the query that the user added or the agent is looking for information for. So now it comes back to us and say, "How do we deal with this context? How do we make sure that the context that ends up to the main agent is relevant enough?" And this is where evaluation comes in. And this is something that is, again, we talked about this a bit earlier, but this is this balance of the smallest possible amount of context that we can give to the user, which leads to the best results in a way. So yeah, we're trying to dig into that through evaluation. We're doing a lot of iterations, how each piece of context that came up from search was used inside an Assistant conversation. (00:29:46): How this context ended up being used inside a system. How was discovery decreased in a sense? Because if we don't have this embedding, this context surfacing, Assistant needs to and find this context itself. So it will use Prometheus, it will query Prometheus like crazy, it will query Loki, it will query different data sources. So we are trying to minimize this initial discovery by just providing this context upfront. So yeah, there are a lot of iterations that we've done so far and we're still doing that. This is an active process. We're still trying to make this as efficient as possible, but it's a huge bit of context engineering here. Yeah. Nicole (00:30:31): Were there some issues that you ran into while you were building this? Right now I'm thinking of... Okay, I guess that's a yes, but I'm thinking also what do you even optimize for?You're talking about evaluating it. I don't even know what is good. Is good as cheap as possible? Is good as fast as possible? Is it as accurate as possible? How do you score accuracy? Ioanna (00:31:00): So a lot of issues came up during Assistant Search. One of the biggest issues that I had was what is relevant context? What is a relevant resource? What is something that Assistant wants to see? And obviously you have some very dashboard panels, like queries inside dashboard panels is very obvious, right? Alert is very obvious, but what else is there that could be useful for the agent? I think that this was one of the very first challenges, and this is still ongoing. We want to find out more resources that could be very useful for an agent, for Assistant. And then as you said, it goes down to evaluation. How do we evaluate this? Obviously when you talk about embeddings is a separate thing, putting embeddings in a storage DB and evaluating the results that are coming back in these searches is one big evaluation. You need to evaluate recall, precision, are the results that I'm getting but relevant to the query? (00:32:01): Am I getting all the results that I was expecting? These are very deterministic evaluations that we can do and we need to do and we are doing. And then you have the agent side, which is a bit more non-deterministic now because you have all this context that is surfaced in Assistant and you need to evaluate was this context useful? Was this context used even from Assistant? Because one of the things that I've seen and very recently was that we're giving too much context from Assistant Search to Assistant and then Assistant was just doing too much in order to process this context. And in the end it was like, I don't really want to process this context, so I'm going to just do the discovery myself. So you were giving a lot of context into Assistant and then it was not even used. So a lot of tokens were used upfront, but then Assistant was doing its own discovery as well. (00:32:53): So yeah, there are definitely a lot of small things that we need to evaluate here. And the biggest one for me is obviously you want to reduce costs, but I don't think that this is priority one at the moment. Sorry for saying this. You want to reduce time because if you're dealing, recently I hung out with a colleague in Grafana and they said that, "You know what? I know my dashboards, I know my metrics, and if I ask something, I have a skill and I'm giving this skill to Assistant, investigate this alert." And it was taking too much time to do the discovery that it was like, "Okay, I have the metrics. I can do it myself. You are slower than me." And obviously he's a very experienced engineer. He knows exactly what he's doing. But again, you want to beat that, you want your agent to be useful in these scenarios. (00:33:45): So time, I think to reducing this discovery, this initial discovery is one of my silver metrics. I don't know if you, Ben, have a different opinion, but giving valuable context quickly, Ben (00:33:58): I think Ioanna (00:33:59): It's key for me. Ben (00:34:00): Yeah, definitely. Definitely. I think as the harness has evolved and it's up to the search tool, in my opinion, to provide the best things that it can. And then the harness will trim that down later if it needs to. It seems that... Yeah, I agree that I would happily spend a little bit more to save five minutes of time of the Assistant searching loads and loads of dashboards in context when otherwise I wouldn't use it and I would just try to provide that context myself, which is what we're trying to avoid by providing all these tools. Nicole (00:34:38): I think that it's also so dependent on your use case, right? Because if I'm in the middle of an outage, give it to me as quickly as possible. I don't care how much it costs. Really just tell me the answer right now. (00:34:52): But if I'm in a more exploratory mood or maybe if I'm learning something, maybe I'm a Loki expert and I don't know about Tempo, take some time and I'm not in so much of a hurry. I'd rather that it be really accurate because as a beginner, I need things spelled out and I don't want you to assume that I know things. I want you to explain it to me and I'm going to be maybe frustrated if you give me instructions and then I find out I can't follow them for some reason. I might just bounce because I'm so new, I won't even know what I'm doing wrong. So that's difficult to... The ideal solution would be to be able to go between these modes, but users aren't often good at telling agents that this is my priority for this session. Ben (00:35:48): Yeah. And it's something you do see with coding agents, they often have a fast mode on them. If you urgently need to get something out, let's ship the fast mode. Maybe that's something we could consider for Assistant to... I don't know whether we can make use of that fast mode and the underlying models or we'd change our tools to return slightly different amounts of context. Who knows? Nicole (00:36:13): Yeah. The plan mode is kind of like that too, right? In a different way. If I do slash plan, it's like do not use any tools. I don't want you to actually be agentic right now. Yeah, I just want to talk. Ioanna (00:36:32): Yeah. And I think that one of the initial things that we were thinking about when we talked about the surfacing context in general was this onboarding journey for new people in a company. So you have someone that is joining a team and you want them to have a buddy on this journey. So having a system, being able to surface all this context for a new joiner, I think that it's quite valuable. And we have inside the system search, in general, in this context layer inside Grafana, we have something that is called infrastructure memories as well. (00:37:14): Obviously infrastructure memories can be helpful in a lot of scenarios. The usage is extremely wide, but one of these was a new joiner. Imagine that you have a team that doesn't have very up-to-date diagrams because this happens, you are in a team and you don't really update. You have the service and you never put it on your diagram. You want to have this live, getting service live inside Grafana and infrastructure memories help into that as well. So yeah, having a persona, like a new joiner, it's quite useful I think even for evaluating purposes as well. Do we help someone that with very little context to get onboarded quickly? Nicole (00:38:02): And does Assistant Search use tools or search through the information available through tools by default? Ioanna (00:38:11): So it doesn't use tools itself. So it just goes and queries the different parts of our DBs to find all this information. At the moment, we're experimenting with a layer between Assistant Search tool and Grafana system. So we can have a condensed version of all this context to be delivered to the main agent. And this is for context compression as well. But yeah, it's not calling any tools at the moment. But yeah, there are other tools like infrastructure memories, something that is exposed to search, but it's also using Assistant Search to find relevant metrics from dashboards, from other tools, et cetera. So it's used by other tools, but it's not using tools at the moment. Yeah. Nicole (00:38:58): So when we talk about context, so far we've been talking about information that you might already have about your stack, you might already have panels and metrics and telemetry and stuff, but another part of context that also goes into the context window are tools. And I saw this on the docs of Claude actually, it says that on average, they're saying that a typical multi-server setup... Hey look, we got to mention Grafana there. Already has 55,000 tokens in definitions before Claude does any work. We were talking about a 200,000 context window in the early days. 55, so a quarter of it is already being used before you even said anything. Ben (00:39:52): Yeah. Nicole (00:39:53): Yeah. And the tool search reduces this by over 85%. Ben (00:39:59): Yeah, which is - Nicole (00:40:00): So Ben (00:40:00): Just by being Nicole (00:40:01): Smart. Ben (00:40:02): Exactly. And it's game changing in fact for the way that MCP servers are developed and tools can be exposed, you can suddenly worry a lot less about how many tokens your tool definitions expose as long as you can assume that user using a sensible harness like Claude Code or Codex or Copilot because they are smart about how they search for tools and that they actually do search for tools rather than just upfronting the cost to the agents. I wonder if I can quickly share my screen and I would like to have an example Nicole (00:40:39): Of this, Ben (00:40:40): But I don't know whether... Yeah, I just want to quickly share what my Claude Code now says when I do slash context. Can you see my Claude Code window there? Nicole (00:40:53): No, we see the stream. Can you Ben (00:40:56): Share the stream? Oh no, that's the Nicole (00:40:58): Word. It's okay. Ben (00:41:02): Okay. Nicole (00:41:03): Well, it's nothing private. People are already seeing that. Okay, we can see it. What are we Ben (00:41:09): Looking Nicole (00:41:09): At? Ben (00:41:10): If you run /context in Claude Code and the other agents, you get this nice overview of your context usage so far in the current conversation. And it tells you here what is taken up by different things such as system tools, MCP tools, MCP server instructions. And the amazing thing is here, MCP tools are now zero tokens to begin with because they're not loaded until they're needed. It's only later on when the agent decides to search for tools that they become loaded. So we're down from 55,000 to almost zero until loaded. There's a few instructions there, which is pretty great. Ioanna (00:41:49): Six months ago, this was dramatically different. Ben (00:41:54): Yeah. And I would love to have an old version of Claude Code installed so that I could run it and see what would've returned then. Ioanna (00:42:02): I might have some screenshots. It's the same that I didn't search for them. But because I had a go with dynamic discovery at the time, I was trying to minimize this as much as possible. And I think that they have some screenshots locally. Nicole (00:42:19): What do we do for the Assistant now? I think, was it Cyril that said it in one of the previous live streams? I think he said that we were... I don't know, maybe it was six months ago, but I think he said we were at 40,000 for system prompts on Assistant. Ben (00:42:41): We're significantly lower than that now. I think it's closer to 15,000. I think he did tell me the other day and I forgot on again. It changes regularly, but yeah, it's a lot lower than it used to be now, considerably lower. And we've noticed... And that's possible because the models are smarter, so they can do a lot more without quite so many instructions. So we also do use deferred tool loading. Many of our tools are deferred rather than loaded up front because they're not needed in 90% of conversations. And we have a couple of strategies for that. We make use of the two different types of tool search, which is the regex search and a semantic-y style search for tools. And I think there's now... We've recently released another method of doing tool search, which we control ourselves, which is... I haven't looked into the implementation details of that, but it's a similar thing to what Claude Code does where it will search through the description and the name and the parameters to find the relevant tools when it needs them. Nicole (00:43:52): So when you say tool search, you're saying that you're giving the Assistant the means to search through tools instead of just already saying load them all up. Ben (00:44:03): Exactly. Yeah. Yeah. So they're defined, they're available to it. It knows which tools are there by name perhaps or by description, but it doesn't know all of the details around the tool, a full parameter spec, which ends up being huge because it's a big JSON schema object to our token saved there. Nicole (00:44:23): I also like this kind of framework by LangChain. These are their four strategies for context engineering and they separate it out into write, select, compress and isolating context. And you're talking about trimming the context to remove irrelevant tokens. So don't give all of the context, don't give all the tools upfront. And it also says summarize context. So you said that the system might know kind of what the tool does, but it doesn't have all of the details. Is that what you meant that it's kind of summarizing? Ben (00:45:00): Everything we're talking about here probably comes under the select context section. Into the select. Nicole (00:45:05): Okay. Ben (00:45:05): Retrieving the relevant tools. So the compression part I think comes in two places. First of all, when you do a slash compact command and you want to compact your session down into something smaller and maybe give some instructions on how to do that, in which case often it will completely summarize the conversation or perhaps it will just trim out the output of tool calls, which will save you quite a lot there. Another way this comes up is in subagents. So if you have a subagent that's spawned to go and perform a specific task, whether that is explore a code base or research GitHub issues or something, it will have a full context window to do that. But what it returns at the end will be a summarized version of all the research that it did. So it's not like you have a whole separate context window which is then summarized and put into your main context window. Ioanna (00:46:04): Why does it Ben (00:46:04): Start - across multi-agent there. Go on. Ioanna (00:46:09): To add to that here, on the compressed context one is what we talked about in Assistant Search as well. This is what we're expanding at the moment, adding a subagent that would be able to grab all this context from Assistant Search and just give back to the main agent the necessary inform only. So Ben (00:46:29): Yeah, Ioanna (00:46:31): I think these are the use cases here. But yeah, deferring tools and it's as Ben said on the select context side. Nicole (00:46:39): So what is tool deferral? Ben (00:46:42): That is the idea of rather than providing the full definitions of the tool upfront, you just provide some Nicole (00:46:50): Small Ben (00:46:50): Description. Yeah, probably the name or perhaps some instructions from the MCP server or maybe the description. And it's very closely related to this idea of progressive disclosure where the agent will be able to start off with just the names of the tools and then eventually it can call a load tool description tool, which increases its context about each individual tool a little bit and then it'll go through the process of then loading the full tool. So it goes through several steps before it has access to the full tool definition to save space in the context window there. Yeah. Nicole (00:47:29): So can you talk a bit more about progressive disclosure? Ben (00:47:34): Yeah, I think this is a really important concept that was developed, however, maybe a year ago now, I think it was first announced. And prior to the idea of progressive disclosure, we just had everything shoved into the context window straight away. So you had all of the skills and all of the tools and all of the rules and everything was just in there. So you would start off with 55,000 tokens of tools, you'd have however many tokens of skills, and before you even started your session, you'd used up 100,000 tokens already, half of your context window at the time. And now rather than doing that, the agent is able to progressively disclose the details of each of those things. It can load full skills on demand using its own load skill command. It can load tools dynamically, and it can do the same thing with other parts of context as well, whether that's things like memories come into that as well. (00:48:44): It's the mechanism by which deferred tools and deferred skills are loaded into the context. Nicole (00:48:53): Okay. So we've talked about ways to reduce the context, especially related to tools or reducing the tools in the first place. I think in the docs that I showed from Claude, they said that... Maybe I can show it again actually. They said that Claude's ability to pick the right tool degrades once you exceed 30 to 50 available tools. So just reducing the number of tools is one strategy. Searching over tools instead of listing them, progressive disclosure and tool deferral. What is this here in the select context from LangChain? It says retrieve long-term memory. What does that involve? Ben (00:49:42): This is an interesting one. I think long-term memory in this space, and I think of memories as things that the model has notes that the model has made in the past and stored ready to make it quicker to solve the same problem in the future. So whereas skills tend to be user defined, or maybe you ask your agent to write a skill on how to solve a certain problem, a memory might be something that the model finds during its work and thinks, "Oh, it would be useful to know that in future, that would've saved me a lot of time here." So it will save this memory and then in future when it encounters a task that looks similar based on the description, maybe it'll search for any memories that are related, load them up, and suddenly have a lot more context on how to solve a specific problem. (00:50:34): And I guess the idea is that you frequently solve the same problems over and over again. In our case, we're bumping memory limits or CPU limits, and it's useful for the model to know exactly where in your code base that is done, but it's not always the sort of thing you think to create a skill for. Instead, the model can do it for you nowadays. Nicole (00:51:02): Okay, cool. There's also... I mean, I don't really mean to be showing so much from Claude and Anthropic, but they write really cool blogs. Ben (00:51:13): Yeah. No, they do. They're doing Nicole (00:51:14): A really good job on that. Also, they have a lot of data because they have a lot of users, and so they can look through the usage patterns, and I find that really useful. One surprising thing here in this article, which I'll also post, is that they removed 80% of Claude Code's system prompt, and they thought that they saw no measurable loss in terms of evaluations and the quality of the responses. So how is that possible that they can remove 80% of the system prompt and still have no effect from that? Ben (00:51:58): I don't know if you've read many system prompts, as you're not an AI, I would assume not, but at least in the past, they were very verbose. Ioanna (00:52:10): Very verbose, yeah. Ben (00:52:12): And they used to repeat things a lot. You used to have to say, "Do not do this." And we would repeat, "Make sure you don't make any mistakes over and over again." These kind of things that are kind of obvious and you don't need to say them to a human. And we'd often find that the system prompt would include lots of details that if you're even remotely familiar with an area, you already know. And so because models back two years ago weren't very smart, you'd have to include this information because models wouldn't know all these things. Nowadays, they are very intelligent and they know a lot more. They've been trained on a lot more data and all of the training strategies mean that they are often more familiar with various concepts than humans are. So you don't need to be quite so explicit in telling them what to do and what not to do. (00:53:07): The idea is they will figure it out because they're resourceful and clever. So all of the previous instructions can just be deleted Ioanna (00:53:17): And Ben (00:53:18): Work. Ioanna (00:53:20): Ben, correct me if I' wrong, but I think that we've done iterations like this inside the Assistant team as well. We've kind of reduced context prompt (00:53:28): In some cases, though it's a very difficult exercise and it's something that you need to be very confident at your evaluation framework to say this and say, "Yeah, you know what? I'm going to rip off 80% of my prompts and be confident that nothing will address." I think that this is a very, very difficult exercise. And also just keep in mind that obviously this is still evolving and I'm pretty sure that our iterations will be a bit more aggressive going forward on reducing the prompt. But at the same time, agents are writing code at the moment. So there is a lot of information that gets repeated that on a PR review, maybe you're going to miss it. I'm sorry for saying this, but this has actually happened to me. So you're adding parts of the prompt that are potentially mentioned somewhere else in the system and you need to make sure that this will be removed afterwards. Nicole (00:54:26): There are a couple of... Sorry. There are a couple of good accidents as well. I think there have been times when people in the history of Assistant have accidentally deleted the system prompt and it's like, "Oh, our evals didn't even notice. Why not?" And then there was one time when someone as a gag changed the prompt to say, "Now speak like a dog," or something like that. And then it was still really good. So I think those - Ben (00:55:01): I don't want to say the scores went up, but I'm pretty sure they didn't go down by that much. Nicole (00:55:06): It's just Ben (00:55:07): Crazy to think. Yeah. Nicole (00:55:09): Yeah. I think it changes as well with each model. And I think that's the whole reason why we now have prompt analysis and Agent Observability, because we started seeing that the prompt was something that also needed to be tweaked with every new version release of whatever model we're using. And it's like there's just so much to keep track of. Ben (00:55:33): And Nicole (00:55:33): I think it's interesting that in this article, I actually like the stuff here on the side, even if you don't read the article, it's saying before then we put it all up front, now we use progressive disclosure and it kind of mirrors what we've seen in Assistant as well. Recently, I saw that we deferred tool definitions and the context footprint was cut by 70%, but I thought it was interesting that it also increased the number of turns, the reasoning turns by 40%. So it's not good in all aspects. Ben (00:56:17): Yeah. There's no silver bullet there as always, is there? You're going to accept some cost at the retrieval stage because you haven't upfronted all of those tokens. I guess you hope that the advantage is on... And you don't just hope, you measure these things that yes, it takes more turns, but the total token usage is lower and the wall clock time is lower because each turn takes less time because there's fewer tokens in the context. Yeah, there's just a lot of different trade-offs to make when you do these things and you just need to make sure that you are measuring correctly, which is surprisingly hard to do. It's really easy to fool yourself into thinking that this is going to make a big difference and without tracking the right metrics in your evals, you're not actually getting the proper view of what's happening. Yeah. Nicole (00:57:10): So what's next in the future of the Assistant when it comes to context? What are the next few problems that you want to tackle at least, if not solve? Ioanna (00:57:22): Interesting question. Yeah, for me, it's more evals. Make sure that we have a very good framework around that context. I think that we are almost there, but models are evolving, things are changing and we need to keep up, and this is a big exercise by itself. So yeah, definitely on the Assistant Search side, surfacing more information, make sure that the context on the diagram that you had before, make sure that the context that we're giving to the main agent is just enough. And this is, again, something that needs very good valuation. So yeah, provide more valuable context. It's definitely something that we need to keep pushing for. Decreasing discovery, make sure that the conversation ends quicker and more efficiently for the user, less expensive, all this. Ben (00:58:20): Yeah. Another thing I think we're finding is that we have an ever expanding product suite and we want the Assistant to be able to do everything that a user could do inside Grafana. And currently the approach is that each team contributes their tools so the Assistant can make use of their products. But the way things are going, models are very good at figuring out when to use tools now. So it's very possible that in a year, the approach to writing tools will be very different. And rather than having use case specific tools, instead you do just offer a way of making API requests against something with a spec, like an OpenAPI spec, and the Assistant makes effectively curl calls against an API and achieves the same results with much less work to constantly keep those tools updated and make sure that they are being defined correctly. (00:59:19): We've seen cases where just adding an API request tool to an Assistant can make it really effective, especially at solving that kind of long tail of tasks where you don't have a specific tool to find. Instead, you need the agent to be able to figure out how to make the right request. And as long as it has the tools to look up the API spec and make that API call, it's possible. So that could be the direction that things are going. Who knows? Nicole (00:59:48): Yeah, that seems to be a trend that just because AI is super powerful and we have all these agentic tools at hand, but actually some of the most efficient ways of retrieving information we've already built. I like the idea that it becomes a problem of knowing when to use AI and when not to, and that's actually the most efficient way to do things. Just to finish us off, I do want to mention ObservabilityCON is on. It's going to be on October 19 to 21 in San Francisco. I will not be there because it's in the US and I'm avoiding it, but lots of people are going to be there and there's going to be a lot on AI and observability. There's going to be workshops, lots of people will be talking about along the same lines of Assistant and we're going to have some announcements. (01:00:49): We're going to be talking about Assistant Search as well. So there's still time to sign up for that if you want. I want to thank you both for joining me today and teaching me all about context. And thank you everyone who's watching this now and after the fact as well. Ben (01:01:07): Yeah, thank you very much for having me. Ioanna (01:01:09): That Ben (01:01:12): Was really good. Cheers. Nicole (01:01:13): Yeah, I'm glad you both could be on and maybe will come back too, just like Ben. Ioanna (01:01:20): Yay. It was fun. Nicole (01:01:25): Great. Another time I want to have someone to talk about compaction of context, but that's for another episode. Thanks everyone. See you