%%
date:: [[2024-04-08]]
parent::
%%
# [[Loki Community Call 2024-04-04]]
![[Loki Community Call 2024-04-04.svg]]
[on YouTube](https://www.youtube.com/watch?v=lwaT6LkMt8c)
speakers:: "[[Owen Diehl]], [[Ed Welsch]]"
00:00 Intro
01:29 What are bloom filters?
09:02 Explore Logs
11:13 Native OTel ingestion endpoint with structured metadata
13:53 Helm charts
16:17 Lots and lots of cleanup
23:00 Quick demo of Explore Logs
- Bloom filters in external preview for 3.0
- It works but it's not fast or cheap or perfect yet
- Still experimental for now
- Been working on this for months now
- Bloom filters are a space efficient probabilistic data structure. They mixed this with N-grams (sequential set of characters), which lets them index/store everything in log lines in bloom filters and then reconstitute queries/filter expressions. You can pre-filter out chunks that you won't need to query because they don't match it.
- Basically lets you do a full text "needle in haystack search", like looking for a support UUID or transaction ID
- This is good for more exploratory searching-- if you know the date range or if you know more metadata, that's still the best way to do it. If you don't, though, bloom filters would be the way to go.
- Bloom filters tell you where *not* to look in the first place.
- This is a way to do query acceleration without introducing schemas, which is kind of core to Loki-- not introspecting too much right away and having a schema.
- Explore Logs
- Queryless experience for exploring logs
- Can't be shown until GrafanaCON
- Displaying patterns in log lines and showing them, visualising a lot of aspects of logs.
- "Structured metadata"
- Native [[OpenTelemetry]] ingestion endpoint and better OTel user experience query experience
- Right now, if you use Loki with [[OpenTelemetry Collector]], the OpenTelemetry metadata is more expansive than what Loki was built for. That was getting serialised into the log line and then you have to parse it. Now, there is a place for structured metadata so that that stuff doesn't have to go either in the index or in the log line.
- Native OTLP (OpenTelemetry Line Protocol) endpoints for Loki itself in 3.0 so you can write it directly to Loki.
- Helm chart
- Used to use [[Jsonnet]] but it hasn't had as much adoption.
- Updated to support microservice mode, including memcache
- Goal is to replace everything built with Jsonnet with Helm. [[Grafana Tanka]] vendors and deploys Helm which is a good way to use Helm. We will deprecate the other 4/5 Helm charts we've been using over the years.
- Cleanup
- General code cleanup
- Renaming metrics
- Better defaults
- Focus on [[Time-series database|TSDB]] index and v13 Schema and deprecating external indices like [[Cassandra]]
-
%%
# Text Elements
%%
# Drawing
```json
{
"type": "excalidraw",
"version": 2,
"source": "https://github.com/zsviczian/obsidian-excalidraw-plugin/releases/tag/2.1.1",
"elements": [
{
"id": "V7hRAFau",
"type": "text",
"x": 388.5514131784439,
"y": -180.92675018310547,
"width": 3.8599853515625,
"height": 24,
"angle": 0,
"strokeColor": "#1e1e1e",
"backgroundColor": "transparent",
"fillStyle": "solid",
"strokeWidth": 2,
"strokeStyle": "solid",
"roughness": 1,
"opacity": 100,
"groupIds": [],
"frameId": null,
"roundness": null,
"seed": 1628134616,
"version": 2,
"versionNonce": 1212942040,
"isDeleted": true,
"boundElements": null,
"updated": 1712048641036,
"link": null,
"locked": false,
"text": "",
"rawText": "",
"fontSize": 20,
"fontFamily": 4,
"textAlign": "left",
"verticalAlign": "top",
"containerId": null,
"originalText": "",
"lineHeight": 1.2
}
],
"appState": {
"theme": "dark",
"viewBackgroundColor": "#ffffff",
"currentItemStrokeColor": "#1e1e1e",
"currentItemBackgroundColor": "transparent",
"currentItemFillStyle": "solid",
"currentItemStrokeWidth": 2,
"currentItemStrokeStyle": "solid",
"currentItemRoughness": 1,
"currentItemOpacity": 100,
"currentItemFontFamily": 4,
"currentItemFontSize": 20,
"currentItemTextAlign": "left",
"currentItemStartArrowhead": null,
"currentItemEndArrowhead": "arrow",
"scrollX": 583.2388916015625,
"scrollY": 573.6323852539062,
"zoom": {
"value": 1
},
"currentItemRoundness": "round",
"gridSize": null,
"gridColor": {
"Bold": "#C9C9C9FF",
"Regular": "#EDEDEDFF"
},
"currentStrokeOptions": null,
"previousGridSize": null,
"frameRendering": {
"enabled": true,
"clip": true,
"name": true,
"outline": true
}
},
"files": {}
}
```
%%