# [[GOH 29 - AI Observability with Grafana with Ishan Jain]]
[in developer-advocacy](obsidian://open?vault=developer-advocacy&file=projects%2FGrafana%20Office%20Hours%2FGOH%20-%20LLM%20Observability%20with%20Ishan%20Jain)
![[GOH 29 - AI Observability with Grafana with Ishan Jain]]
<iframe width="560" height="315" src="https://www.youtube.com/embed/9X7M-bvnZG0" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
Related:: ""
## Title
- guide to llm observability (47)
- llm observability (37)
- intro to llm observability (46)
- ai observability (68)
- intro to ai observability (52)
- guide to ai observability (33)
## Talking points
- Intro
- *Hello and welcome to Grafana Office Hours. Artificial intelligence has gone very quickly from something out of sci-fi books to something quite commonplace that we use in various forms as part of our every day lives. Some of us might have even tried our hand at creating apps leveraging AI SDKs. Our guest today is going to talk about observability for AI: why we should do it, what to watch out for, and how exactly to set it up.*
- *I'm Nicole van der Hoeven, a Senior Developer Advocate at Grafana Labs, and I'm joined by a colleague of mine, Ishan Jain, a Senior Developer Experience Engineer.*
- Introduce guest: Ishan Jain
- Who are you?
- What do you do?
- Definition of terms
- Artificial intelligence
- Machine learning
- LLM
- What are we talking about observing today?
- Ishan says AI is the biggest area. ML is under AI and LLM is under ML.
- Why should we monitor AI?
- Is AI Observability just another type of Application Observability?
- How are AI-based apps different?
- massive datasets
- costs can increase quickly
- possibility of model drift
- security concerns
- existence of rate limiting for many models
- latency is important
- What do we need to monitor?
- Traces
- Request metadata, like temperature (amount of creativity/randomness), model version, etc
- Response metadata, like tokens and cost
- These are the most important for LLMs becuase it talks about the sequence of events. Did I make an LLM call or did I make a call to the Vector DB instead?
- Metrics
- Request volume
- Request duration
- Costs and token counters
- What about logs?
- How can we monitor AI?
- Instrumentation
- Manual: OpenTelemetry
- Automated: OpenLIT SDK, Traceloop, Langtrace?
- Demo? : AI integration for Grafana Cloud: AI Observability (Tempo, Prometheus or Mimir)
- There was already an OpenAI integration but it wasn't OTel specific. He really wanted something that was a one line or one-click thing.
- OpenLit dashboard for Grafana Cloud
- Outro
- If people want to learn more about this topic, where should they go?
## Timestamps
00:00:00 Introductions
00:04:08 What is AI? What are LLM and machine learning?
%%
# Text Elements
# Drawing
```json
{"type":"excalidraw","version":2,"source":"https://github.com/zsviczian/obsidian-excalidraw-plugin/releases/tag/2.0.25","elements":[],"appState":{"theme":"dark","gridSize":null,"viewBackgroundColor":"#ffffff"}}
```
%%