# [[All about OTel and Logging on Kubernetes with Loki - Loki Community Call April 2025 at KubeCon EU]] [in developer-advocacy](obsidian://open?vault=developer-advocacy&file=projects%2FLoki%20Community%20Calls%2F2025-04-03-monitoring-k8s-with-otel-and-loki) ![[All about OTel and Logging on Kubernetes with Loki - Loki Community Call April 2025 at KubeCon EU]] <iframe width="560" height="315" src="https://www.youtube.com/embed/27ZHOg-LaM0" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> Related:: "[[OpenTelemetry|OTel]], [[Kubernetes]], [[Grafana Loki|Loki]], [[Cyril Tovena]], [[Jay Clifford]], [[Ward Bekker]]" ## Titles - otel and logging (87) - collecting kubernetes logs with otel (46) - opentelemetry and logging (24) - logging and otel (57) ``` All about OTel and Logging on Kubernetes with Loki (Loki Community Call April 2025) ``` ## Description ``` In this Loki Community Call, we talk all about OTel and logging on Kubernetes with Cyril Tovena, Ward Bekker, Jay Clifford, and Nicole van der Hoeven at KubeCon EU 2025 in London. We discuss when why you should switch to OTel and why you shouldn't, what OTLP is exactly, and best practices for ingesting data through an OTLP endpoint. TIMESTAMPS: 00:00:00 Introductions 00:02:06 What is OpenTelemetry? 00:04:29 Reasons to switch to OTel 00:09:09 Reasons not to switch to OTel 00:16:17 What is OTLP? 00:19:36 Best practices for ingesting data through OTLP endpoint 00:22:25 Migrating from Loki exporter to OTLP endpoint 00:23:50 Different approaches for getting logs from Kubernetes clusters 00:27:33 k8s-monitoring Helm Chart Office Hours playlist: https://www.youtube.com/playlist?list=PLDGkOdUX1Ujqzknuz670JCP_zMJLRaYPC 00:28:25 Future changes on the horizon for OTel RESOURCES: k8s-monitoring Helm Chart Office Hours playlist: https://www.youtube.com/playlist?list=PLDGkOdUX1Ujqzknuz670JCP_zMJLRaYPC Loki docs: https://gra.fan/lokidocs 📢 Have questions? Submit them here: http://youtube.com/post/UgkxUASXEj6hOoHSzQu7nD_YirrIst_RLYoj?feature=shared ``` ## Talking points - Announcements - We're at KubeCon! - Introduce guest - Who are you? - What do you do? - How long have you been using Grafana/other project? - What is OpenTelemetry? - Why is it needed/why it's important (no vendor lock-in) - Got popular for tracing first. Then added metrics and now logs. Also supports [[Baggage]] ([OTel docs on this](https://opentelemetry.io/docs/concepts/signals/baggage/)). - Proposal phase: profiles, events - Archived: [proposal to support](https://github.com/open-telemetry/oteps/issues/169) [[Real User Monitoring|RUM]] events in OTel. - [Talk at KubeCon EU 2024](https://www.youtube.com/watch?v=l2_wsvv-Rhs) by [[Purvi Kanal]] from [[Honeycomb]] about it. - [[Grafana Beyla|Beyla]] was donated to [[CNCF]]. - How should you decide whether to use it or not? - Pros - Easier to get started because of best practices already baked into OpenTelemetry-- you don't have to think of what labels to use, and Loki supports OTel metadata natively - No vendor lock-in - With the OTel Collector, you can also have log pipelines to redirect some logs to, for example, security ([[Seam]]) - OTel is very heavily supported by the community and is being actively developed by a lot of people - Cons - Some log types are not yet available in OTel (like specific router types) - It can take time and effort to switch to OTel - Some people don't have an always-on connection to the database - Doesn't support filtering or transforming - Performance overhead - it's heavier than things like fluentd + Fluent Bit (but Alloy does do some to improve performance) - Factors to consider - Complexity - Futureproofness? - Performance overhead - Use cases (infrastructure) - Alternatives - Prometheus (also has a native OTLP endpoint) - [[fluentd]], [[Fluent Bit]] - OTLP endpoint in Loki 3.0 - OTLP is the protocol definition for sending telemetry data - you no longer need a transformation layer to convert data to this format. Previously this transformation was done using the [Loki Exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/lokiexporter/README.md) - now deprecated! - Docs: https://grafana.com/docs/loki/latest/send-data/otel/native_otlp_vs_loki_exporter/ - Making Loki a first-class citizen in OTel - Having the OTLP endpoint is a good step but we're not there yet. - OTel is still different from Loki and it's not 1:1. Loki is just logs and labels, and OTel has logs, labels, and tons of attributes. Need to make UI better so that it doesn't hide the metadata. [[Drilldown Apps]] also could be improved on thsi front. - Demo - Best practices in ingesting data through OTLP endpoint. What to put in structured metadata? High cardinality? - Loki has a default config that does the right thing automatically. OTel has log attributes and resource attributes (low cardinality). The resource attributes are mapped to Loki stream labels that also are low cardinality - we only want up to 15. - Then the log attributes (like transaction ID, user ID, trace ID) are mapped to structured metadata. That way we can support them in a high cardinality fashion so we don't need to index them. If you use OTel best practices, Loki will work well with that. - You can also add additional attributes - like team-based attribution is a common use case. - Migration (Loki exporter to OTLP) - Loki 3.0 required - configuration - point OTel logs at Loki - Landscape of log collection for Kubernetes clusters - Different approaches for getting logs from k8s pods - Cyril heard of a new approach that doesn't involve scraping the files - OpenTelemetry's work on getting pod logs in an OTel compliant way - [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator) and [how it works with Grafana Cloud](https://grafana.com/blog/2024/10/17/how-to-quickly-configure-grafana-cloud-application-observability-with-opentelemetry-operator/) - Different approaches for getting logs from Kubernetes clusters - OTel can collect podlogs directly from Kubernetes and can transform them - Kubernetes event receiver to be better able to alert on events - [[Grafana Alloy|Alloy]] has more features than OTel collector like syslog receiver. Enhances standard collector, still supports [[Prometheus]]. [[Promtail]] was integrated into Alloy. - The future of OTel (not just logs) - [[eBPF]] profiling - OTel Semantic conventions ## Timestamps 00:00:00 Introductions 00:02:06 What is OpenTelemetry? 00:04:29 Reasons to switch to OTel 00:09:09 Reasons not to switch to OTel 00:16:17 What is OTLP? 00:19:36 Best practices for ingesting data through OTLP endpoint 00:22:25 Migrating from Loki exporter to OTLP endpoint 00:23:50 Different approaches for getting logs from Kubernetes clusters 00:27:33 k8s-monitoring Helm Chart Office Hours playlist: https://www.youtube.com/playlist?list=PLDGkOdUX1Ujqzknuz670JCP_zMJLRaYPC 00:28:25 Future changes on the horizon for OTel ## Resources k8s-monitoring Helm Chart Office Hours playlist: https://www.youtube.com/playlist?list=PLDGkOdUX1Ujqzknuz670JCP_zMJLRaYPC %% # Excalidraw Data ## Text Elements ## Embedded Files 5ef1c9944534aaf36011317ca44bcf066b514d2c: [[All about OTel and Logging on Kubernetes with Loki - Loki Community Call April 2025 at KubeCon EU.png]] ## Drawing ```json { "type": "excalidraw", "version": 2, "source": "https://github.com/zsviczian/obsidian-excalidraw-plugin/releases/tag/2.9.2", "elements": [ { "id": "lIVWwux0", "type": "image", "x": -153.63602578639984, "y": -34.997833251953125, "width": 500, "height": 281.25, "angle": 0, "strokeColor": "transparent", "backgroundColor": "transparent", "fillStyle": "hachure", "strokeWidth": 1, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "roundness": null, "seed": 5963, "version": 3, "versionNonce": 1938848748, "updated": 1745405045137, "isDeleted": false, "groupIds": [], "boundElements": [], "link": null, "locked": false, "fileId": "5ef1c9944534aaf36011317ca44bcf066b514d2c", "scale": [ 1, 1 ], "index": "a0" } ], "appState": { "theme": "dark", "viewBackgroundColor": "#ffffff", "currentItemStrokeColor": "#1e1e1e", "currentItemBackgroundColor": "transparent", "currentItemFillStyle": "solid", "currentItemStrokeWidth": 2, "currentItemStrokeStyle": "solid", "currentItemRoughness": 1, "currentItemOpacity": 100, "currentItemFontFamily": 5, "currentItemFontSize": 20, "currentItemTextAlign": "left", "currentItemStartArrowhead": null, "currentItemEndArrowhead": "arrow", "currentItemArrowType": "round", "scrollX": 649.4220581054688, "scrollY": 367.1957702636719, "zoom": { "value": 1 }, "currentItemRoundness": "round", "gridSize": 20, "gridStep": 5, "gridModeEnabled": false, "gridColor": { "Bold": "rgba(217, 217, 217, 0.5)", "Regular": "rgba(230, 230, 230, 0.5)" }, "currentStrokeOptions": null, "frameRendering": { "enabled": true, "clip": true, "name": true, "outline": true }, "objectsSnapModeEnabled": false, "activeTool": { "type": "selection", "customType": null, "locked": false, "fromSelection": false, "lastActiveTool": null } }, "files": {} } ``` %%