# [[OpenTelemetry]] ![[OpenTelemetry.svg]] [site](https://opentelemetry.io/) [repos](https://github.com/open-telemetry) OpenTelemetry ("OTel") is an [[Open source]] project to facilitate and standardize the collection of [[Telemetry]] signals for [[Observability]]. It is the merger of two other tools, OpenTracing and OpenCensus. OTel is available in several languages and is a [[CNCF]] project. It's the second most active CNCF project behind [[Kubernetes]]. [^grafana] On **21 May 2026, OpenTelemetry graduated from the CNCF** — the highest project maturity tier (Sandbox → Incubating → Graduated), which signals to adopters that it is mature, well-governed, vendor-neutral, and production-proven. To graduate, OTel had to pass independent third-party security audits (including the [[OpenTelemetry Collector]]) and a fuzzing audit, complete a formal governance review, and demonstrate broad production adoption across many companies (12,000+ contributors from 2,800+ organizations). Graduation reflects the maturity of the *project and its governance* — it does not mean every signal or semantic convention is stable (see the status tables below). <iframe width="560" height="315" src="https://www.youtube.com/embed/qI1aJyShpbk" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> ## Areas "OTel" refers to several things because there are multiple components to the project: - [[OpenTelemetry Specifications]] are standards (including [[OTel Semantic Conventions]]) set by committees about the format of different telemetry signals - [[OpenTelemetry Logging Specifications]] - [[OpenTelemetry Instrumentation SDKs]] are libraries and APIs for implementing OTel in different languages - The [[OpenTelemetry Collector]] is a [[Telemetry collector|Telemetry collector]] for gathering telemetry signals - OTel Special Interest Groups (SIGs) are self-selecting working groups formed to collaborate on specific projects - [[Open Agent Management Protocol|OpAMP]] is a standard for managing [[Telemetry collector]]s. <iframe width="560" height="315" src="https://www.youtube.com/embed/Xhh9V7uRP88" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> ## Ecosystem and key components Beyond the spec, SDKs, and Collector, OTel has grown a broad ecosystem: **Collector and distributions.** The [[OpenTelemetry Collector]] receives, processes, and exports telemetry over OTLP, with no built-in storage. [[Grafana Alloy|Alloy]] is Grafana's opinionated open-source Collector distribution (the successor to [[Grafana Agent]] in "flow mode"); it can run OTel and Prometheus pipelines side by side and aims to be a fully compliant Collector distribution with native OTel YAML support. The all-in-one `docker-otel-lgtm` container bundles the Collector with the LGTM stack for quick local demos. **Agent management — [[Open Agent Management Protocol|OpAMP]].** A standard for remotely managing fleets of collectors/agents; used by Alloy and Grafana Fleet Management. **Auto-instrumentation.** OTel increasingly favours zero-code instrumentation. Approaches range from JVM-level agents (true zero-code, can hook into more than the framework exposes, but more of a black box), to framework-level (e.g. Quarkus, Spring Boot starter), to code-level. The OTel Operator injects instrumentation via Kubernetes lifecycle hooks; the newer `opentelemetry-injector` uses `LD_PRELOAD` so it also works outside Kubernetes (VMs, Cloud Foundry). See [[OTel Autoconfigure]] and [[Declarative Configuration for OTel SDKs]] — configuration moves out of code into YAML/env vars (still experimental for Java and PHP). **eBPF instrumentation.** [[Grafana Beyla|Beyla]] is eBPF-based auto-instrumentation that captures OTel metrics and traces for HTTP/gRPC on Linux, surfacing data other instrumentation can't (e.g. time a request spent queued). It is being donated upstream as **OpenTelemetry eBPF Instrumentation (OBI)**. **Profiles — the fourth signal.** OTLP Profiles reached public alpha (announced at KubeCon, March 2026). [[Grafana Pyroscope|Pyroscope]] has experimental OTLP-profile support. Profiling answers what traces can't: a span shows how long something took, profiling shows *what* consumed that time, via the flame graph. **Semantic conventions tooling.** Weaver generates code and docs from the [[OTel Semantic Conventions|semantic conventions]]; adopting semconv makes instrumentation far more automatic (related: `promconv` for Prometheus). **Efficiency — OTel Arrow.** OTLP with Apache Arrow uses columnar compression for bulk telemetry, with reported gains of ~30% for traces and 50–70% for logs/metrics versus plain OTLP. **OTLP + Prometheus.** Prometheus 3.0 added *resource attribute promotion*, promoting OTel resource attributes (e.g. `service.name`, `deployment.environment.name`) to metric labels — removing older `target_info` join workarounds. **Testing.** [[Tracetest]] enables trace-based testing of distributed apps on top of OTel tracing (works with Jaeger, Tempo, OpenSearch, or anything with an OTel Collector). **AI / LLM observability.** OTel can instrument AI apps, though some AI-specific data still needs custom attributes today; [[OpenLIT]] is an OTel-based framework for instrumenting and evaluating LLMs. See also [[AI Observability]]. **Governance.** OTel is steered by an elected Governance Committee (vision/policy) and a Technical Committee (technical direction), with community SIGs grouped into Specifications, Implementation, and Cross-cutting. ## Why OpenTelemetry? The promise of OpenTelemetry is in the creation of standards and tools that are vendor-neutral. In theory, if an application is instrumented with OpenTelemetry, the company maintaining it can switch [[Observability]] providers (like [[Datadog]] to [[Grafana]] for example) without needing to reconfigure any [[Telemetry]] endpoints. In practice, however, migration from one service provider to another is never easy, and *still* requires significant configuration. OTel is also increasingly the standard for **AI observability**: the [[OTel Semantic Conventions|GenAI semantic conventions]] (`gen_ai.*`) standardise how LLM and AI-agent telemetry is recorded (model, token usage, latency, and — opt-in — prompts/responses), now covering agents, tool execution, evaluation, and the Model Context Protocol (MCP). These were significant enough to be split into their own repository ([semantic-conventions-genai](https://github.com/open-telemetry/semantic-conventions-genai)), though they remain experimental. ## Maturity status of telemetry signals OTel is developed *signal by signal*. Each signal is made of four components (API, SDK, OTLP/Protocol, Collector), so "stable" is rarely all-or-nothing. Status as of June 2026 ([source](https://opentelemetry.io/docs/specs/status/)): | Signal | Status | |---|---| | Context propagation (foundation) | Stable | | Traces | Stable | | Metrics | Stable (API, data model & OTLP stable; SDK still maturing — "mixed") | | Logs | Stable (Bridge API, SDK & OTLP stable; designed as a *bridge* to existing log frameworks) | | Baggage | Stable | | Profiles | Experimental — entered public alpha (March 2026); the emerging fourth signal | | Events | Experimental (built on top of logs) | ## Semantic conventions and their maturity status Semantic conventions define the shared vocabulary (attribute names, span/metric names, units) so telemetry from different tools is comparable and portable. They have their *own* per-domain maturity, separate from the signals. OTel renamed the old **"Experimental"** level to **"Development"**; the lifecycle is now **Development → Release Candidate → Stable**. Status as of June 2026 (from each convention's status badge): | Semantic convention | Status | |---|---| | Exceptions | Stable | | HTTP | Mixed (spans & metrics stable; rest in development) | | Database | Mixed (stable conventions available via opt-in) | | Resource | Mixed (service/SDK/version attrs stable; compute & cloud attrs in development) | | RPC | Release Candidate | | CI/CD | Release Candidate | | Kubernetes attributes | Release Candidate | | Messaging | Development | | FaaS (serverless) | Development | | Feature Flags | Development | | System / host / process / container / hardware metrics | Development | | GraphQL | Development | | Cloud Providers | Development | | Object Stores | Development | | CloudEvents | Development | | Generative AI (`gen_ai.*`, agents, MCP) | Development (experimental) | | Profiles attributes | Development | > Stabilization happens attribute-by-attribute and signal-by-signal. The mechanism for rolling out stable conventions without breaking existing instrumentation is the `OTEL_SEMCONV_STABILITY_OPT_IN` environment variable (e.g. `http`, `database`, `database/dup`). ## Videos I've created on OpenTelemetry ### Grafana OTel Community Call (GOCC) A monthly live stream I co-host on the [Grafana YouTube channel](https://youtube.com/@grafana), where guests talk about OpenTelemetry and Grafana working together. **GOCC 01 — Declarative configuration for OTel** (with [[Marylia Gutierrez]], Oct 2025) — [[GOCC 01 - Declarative configuration for OTel with Marylia Gutierrez|note]] <iframe width="560" height="315" src="https://www.youtube.com/embed/FwnyV5z_ij4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> **GOCC 02 — Lightweight open source APM with the OTel Demo** (Nov 2025) — [[GOCC 02 - Lightweight Opensource APM with OTel Demo|note]] <iframe width="560" height="315" src="https://www.youtube.com/embed/cX74hkdI1zM" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> **GOCC 03 — Logging best practices** (Dec 2025) — [[GOCC 03 - Logging best practices|note]] <iframe width="560" height="315" src="https://www.youtube.com/embed/D-bWeY-gBWU" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> **GOCC 04 — Observability for GenAI applications** (2026) — [[GOCC 04 - Observability for GenAI Applications|note]] <iframe width="560" height="315" src="https://www.youtube.com/embed/LfDVygRWQJU" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> **GOCC 06 — Behind the magic of auto-instrumentation** (Feb 2026) — [[Live - GOCC 06 - Behind the magic of auto-instrumentation|note]] <iframe width="560" height="315" src="https://www.youtube.com/embed/HItumi3qt90" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> **GOCC 07 — Continuous profiling, the OpenTelemetry way** (May 2026) — [[GOCC 07 - Continuous profiling, the OpenTelemetry way|note]] <iframe width="560" height="315" src="https://www.youtube.com/embed/N47wy9pp7Lo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> **GOCC 08 — OpenTelemetry graduation** (22 Jun 2026) — see [[OTel Graduation Livestream Prep]] <iframe width="560" height="315" src="https://www.youtube.com/embed/P9CfVEGwcgo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> ### Other OpenTelemetry videos & talks I've made - [GOH 23 — Getting started with OpenTelemetry](https://youtube.com/live/Xhh9V7uRP88) (with Juraci Paixão Kröhling, Dec 2023) *(embedded near the top of this note)* - [GOH 06 — OpenTelemetry demo app with Grafana, Tempo, Loki, and Prometheus](https://youtube.com/live/dXR8WNm5uos) (Jul 2023) - [GOH 05 — Distributed tracing for testing with Grafana Tempo and Tracetest](https://youtube.com/live/oJ0wdneD8Po) (Jul 2023) - [GOH 28 — OpenTelemetry metrics with Traefik](https://www.youtube.com/watch?v=1Buk3wPpJdY) (May 2024) - [Open source observability explained — the Grafana Labs stack](https://youtu.be/WSW1urIXsfA) (Apr 2024) - [All about OTel and Logging on Kubernetes with Loki — Loki Community Call](https://www.youtube.com/watch?v=27ZHOg-LaM0) (KubeCon EU, Apr 2025) - [Asimov's Zeroth Law of Robotics — Observability for AI](https://www.youtube.com/watch?v=x6EKTCAWtn8) (KubeCon EU 2025) ## See also - [[OpenTelemetry Python SDK]] - [[OpenTelemetry Logging Specifications]] - [[OpenTelemetry and Loki]] - [[OpenTelemetry Instrumentation SDKs]] - [[Logs Drilldown]] %% ## Related - [[OpenTelemetry Logs SIG]] - [[Grafana Labs OTel strategy for databases]] %% [^grafana]: OpenTelemetry OSS | Analyze software performance. Grafana Labs. Retrieved on 23 April 2023 from: https://grafana.com/oss/opentelemetry %% # Excalidraw Data ## Text Elements ## Drawing ```json { "type": "excalidraw", "version": 2, "source": "https://github.com/zsviczian/obsidian-excalidraw-plugin/releases/tag/2.1.4", "elements": [ { "id": "4y8R7iOA", "type": "text", "x": 118.49495565891266, "y": -333.44393157958984, "width": 3.8599853515625, "height": 24, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "roundness": null, "seed": 967149026, "version": 2, "versionNonce": 939059582, "isDeleted": true, "boundElements": null, "updated": 1713723615080, "link": null, "locked": false, "text": "", "rawText": "", "fontSize": 20, "fontFamily": 4, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "", "lineHeight": 1.2 } ], "appState": { "theme": "dark", "viewBackgroundColor": "#ffffff", "currentItemStrokeColor": "#1e1e1e", "currentItemBackgroundColor": "transparent", "currentItemFillStyle": "solid", "currentItemStrokeWidth": 2, "currentItemStrokeStyle": "solid", "currentItemRoughness": 1, "currentItemOpacity": 100, "currentItemFontFamily": 4, "currentItemFontSize": 20, "currentItemTextAlign": "left", "currentItemStartArrowhead": null, "currentItemEndArrowhead": "arrow", "scrollX": 583.2388916015625, "scrollY": 573.6323852539062, "zoom": { "value": 1 }, "currentItemRoundness": "round", "gridSize": null, "gridColor": { "Bold": "#C9C9C9FF", "Regular": "#EDEDEDFF" }, "currentStrokeOptions": null, "previousGridSize": null, "frameRendering": { "enabled": true, "clip": true, "name": true, "outline": true } }, "files": {} } ``` %%