# [[Grafana Loki]] [repo](https://github.com/grafana/loki) | [site](https://grafana.com/oss/loki/) Loki is an [[Open source|opensource]] [[Logs|log]] aggregation tool that takes much inspiration from [[Prometheus]]. It was created by [[Grafana Labs]] . Loki includes a centralized database to forward logs to. This centralization is useful especially in the case of [[Serverless computing]] or ephemeral [[Kubernetes]] pods. Another reason to centralize logs is to see the bigger pictures, especially in the case where you might have multiple servers with [[Load Balancer|load balancing]], such that looking at individual servers would only show you the requests that went to each particular server. Loki works so closely with Prometheus that they use the same label sets as metrics. It also lets you turn logs into metrics to make them easier to work with. Here's a quick intro to Loki:  Here's an introduction to Loki that I made with [[Jay Clifford]]:  ([[What is Loki]]) %% Here's a webinar by [[Aldhair Martinez]] about getting started with logging and Loki:  00:00:00 Introductions 00:02:48 Observability and the Grafana Stack 00:08:18 Demo: Data correlation 00:12:32 Why Loki? 00:21:06 Getting logs into Loki (Ingestion) 00:23:49 Storing and searching logs 00:29:48 Demo: Loki Query 00:36:18 Running Loki (Deployment) 00:41:35 Loki 3.0 features 00:44:58 Q&A %% ## Why Loki? Loki was built to address a few problems: - There wasn't an easy way to get logs out of [[Kubernetes]] with its ephemeral [[Pods]]. There wasn't really service discovery yet. These days, Loki isn't so Kubernetes-specific anymore, but it still works really well with k8s. - Storing logs to disk meant that the disks could run out of space very quickly. Instead, Loki uses [[Object storage]]. - You had to figure out your schema for your logs to be searchable. Loki gets around this by having a very small index and doing [[Schema at query]] instead. - Logs are write-intensive [[Time Series]], so they very quickly get out of hand. Loki is built for efficency and horizontal scalability as well as cost-effectiveness. <iframe width="560" height="315" src="https://www.youtube.com/embed/3uFMJLufgSo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> ## Features of Loki ### Cloud-native Loki was made to work with [[Cloud Native Computing Foundation|cloud-native]] projects, including [[Kubernetes]], [[Prometheus]], [[Grafana]], and more. ### Performant - Loki only indexes metadata instead of the full text, which makes it faster to query and cheaper to run. - Log lines are stored and compressed. - The grepping workload is [[Distributed computing|distributed]] in a [[Parallelism|parallelized]] way. A big problem is broken up into smaller chunks that can be worked on separately. Surprisingly, this parallelized brute force approach leads to performance improvements. - [[Promtail]] uses [[Exponential backoff]] so it doesn't hammer your server. - Only the index (as a sort of table of contents) is placed in object storage, unlike other tools that save everything, including the log line, to disk and to RAM for easy access. - Object storage is 100% persistent: log data is not lost in case of system failures or restarts. This is accomplished through the high availability. ### Horizontally [[Scalability|scalable]] - Loki uses a [[Microservices|microservices-based architecture]]: you can scale up, for example, the query buffer, depending on your needs. ### Object storage Loki doesn't store incoming log lines as rows; instead, it batches and compresses them as streams of text. This enables Loki to store that text in [[Object storage]] rather than more expensive [[Block storage]]. Since object storage is cheaper, Loki is usually more cost-efficient to run. ### Querying options Learning new query languages can be difficult, but highly optimised databases often require their own query language. Loki is no exception: it uses [[LogQL]] as a query language. However, there are a few alternatives that can be used to query Loki without needing to be an expert at LogQL: #### Graphical Query Builder In [[Grafana]], you can create queries for Loki using a visual query builder that lets you choose the labels you want to visualises. The builder then writes the query for you, and you can then optionally modify it. #### Explore Logs In [[GrafanaCON 2024]], a new feature for [[Grafana Loki|Loki]] on [[Grafana]] was announced: *Explore Logs*. Explore Logs is a way to visualize logs, narrow down on errors, and find issues without having to use [[LogQL]]. See: [[How to enable Explore Logs for Loki]] ### "Like [[Prometheus]], for logs" Loki prides itself on being the log counterpart of the metrics-focused [[Prometheus]]. Here are a few reasons why that could be a good thing: - Since [[Metrics]] and [[Logs]] are often both collected as important parts of an [[Observability]] strategy, modelling Loki after Prometheus means platform engineers have less to learn to work with them. - Loki uses [[Promtail]] as a default agent (although you can change that to something else). Promtail uses the same service discovery mechanisms as Prometheus. - Loki works with [[Prometheus Alertmanager]], by letting you create a metric out of your logs using [[LogQL]]. - [[LogQL]] is very similar to [[PromQL]]. ### Push-based Loki is [[Push-based monitoring|push-based]], which means that it waits for logs to be explicitly sent to it. The push-based model is more suitable for situations where multiple components may be sending Loki logs at the same time. Loki can still be coupled with [[Pull-based monitoring|pull-based]] [[Telemetry collector|telemetry collectors]] for ingestion if required. The fact that Loki is decoupled from the ingestor agent allows you to adopt a more modular approach, where you choose the tool that you need instead of being locked into Loki's chosen ingestion agent. ## [[How Loki stores data]] ## [[Architecture of Loki]] ## [[How to install Grafana Loki]] ## [[How to deploy Grafana Loki]] ## [[Ingesting logs into Loki]] ## [[Querying Loki]] ## [[Alerting with Loki]] ## Alternatives to Loki See [[Logs#Tools for collecting logs]] ## Other resources - [Here's a Grafana Office Hours livestream](https://www.youtube.com/watch?v=OLebNPLIJMI) that I did with [[Paul Balogh]] and [[Ward Bekker]] about Loki - [[GOH 27 - Grafana Loki design basics with Ed Welch]] - [[Loki Community Call 2024-04-04]]: Community Call tackling recent Loki features in [[Loki 3.0]]. - [[Loki 3.0]] - [Here's a playlist of all Loki Community Calls](https://gra.fan/lokicclist) that I've done with [[Jay Clifford]]. %% - [[Developer Advocacy for Loki]] - plan for how to improve Loki %% ## See also - [[Prometheus]] - The metrics monitoring tool that Loki was designed to complement - [[Observability]] - The broader practice of making systems observable, of which logging is a key pillar - [[Architecture of Loki]] - Detailed breakdown of Loki's microservices-based architecture - [[Telemetry collector]] - Overview of agents like Promtail that collect and forward telemetry data - [[Instrumentation]] - The practice of modifying systems to expose monitoring information [^goh]: Bekker, W. (2023). *Getting started with Grafana Loki (Grafana Office Hours #09*. Retrieved from: https://www.youtube.com/watch?v=OLebNPLIJMI