# [[Loki Architecture and Components]] <iframe width="560" height="315" src="https://www.youtube.com/embed/_hv4i84Z68s" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> Related:: "[[Grafana Loki]], [[Jay Clifford]], [[Architecture of Loki]]" ## Pre-production %% ### Resources - [Google Doc with Jay](https://docs.google.com/document/d/1XxcddwdrgPPi9ikTYACN8R2NyVpEIEzTr0oPneVBTao/edit#heading=h.pm9egnice656) - [Excalidraw drawings](https://link.excalidraw.com/l/54MVL8XCWbS/8dTw45y0dPw) %% ### Thumbnail ### Title 1. ## Script #### Intro (Nicole) Loki can be deployed in what we call microservices mode. In this deployment mode, Loki is broken up into many different components, each of which can be horizontally scaled if you need to. But before you decide how many instances of each component you need, you'll need to know what each one does. In this video, Jay and I are going to tell you exactly what's under the hood for Loki. We'll break down what every single component is and how they all keep Loki ticking. #### Distributors (Jay) —-------> Break (Jay) To ground our components in a real-world example, we’ll use a log processing facility, treating our virtual logs as real-world logs. Let's start with ingestion. (Jay) —-------> Distributors Whether collected via a collector, SDK, or API, our logs arrive at the door of our log processing facility. This is where our first component, the distributor, kicks in. When lorries of logs arrive, the distributor validates them, ensuring no boulders are included and that the logs comply with tenant or global limits. For example, the lorry can only drop off a maximum number of logs, and the logs must be of a certain length. After validation, the distributor distributes the logs to our next component, the ingesters. This is done via a hash ring… #### Hash ring (Nicole) —-------> Pop up (What a hash ring) Start (Nicole) Hang on, Jay, let's talk about what a hash ring is. You can think of a hash ring as a group of instances of the same type of component (distributors in this case). - Information, in the form of log lines, are sent to this whole hash ring instead of to individual instances. - Then, an algorithm determines which instance gets which data. - That way, if one of the instances is down, or if another joins, then the flow of data continues with minimal disruption. - So a hash ring is kind of like a load balancer, but while load balancers typically distribute incoming traffic or requests, a hash ring distributes data. Okay, Jay, tell us how this algorithm works exactly. #### Distributors continued (Jay) —-------> Pop up (What a hash ring) Stop (Jay) In our case, ingesters join the hash ring indicating they are ready to receive logs. We will discuss exactly what ingesters are next, but for now, imagine an ingester as a production line of debarking and sawing machines. Let's say we have three of these production lines with unique IDs: 0, 25, and 50. The distributor then hashes a log stream, which consists of the tenant and the unique set of labels associated with that log stream. For simplicity, let's say this hashes to 15. Based on the hash ring rules, the log stream will be allocated to the production line with the smallest token larger than the log stream hash, which means production line 25 would get the logs. Now, we need to stretch our imagination a bit. Like most distributed databases, Loki supports replication. Distributors send the same log stream to multiple ingesters to mitigate data loss if one ingester fails. Imagine our log processing plant has a cloning machine that replicates logs according to the replication factor. The hash circle rule applies to the first ingester, but then we move to the next highest available ingester based on the replication value. With a replication factor of 3, our log stream of 15 would go to ingesters 25, 50, and then back to 0. Note that with replication, a quorum of successful writes is needed. For instance, a replication factor of 3 requires 2 out of 3 ingesters to process the logs successfully; otherwise, the write fails, and an error is returned to the distributors. A load balancer is necessary to properly balance incoming traffic to each distributor. In a Kubernetes deployment, this is managed by the Kubernetes service load balancer. Distributors are stateless components, making them easy to scale and offload as much as possible from ingesters. #### Ingesters (Jay) —-------> Ingesters (Jay) Onto ingesters! one of the most critical components within our Loki architecture. As we mentioned earlier imagine ingesters like a production line of debarking and sawing machines. They remove the bark (initial processing of logs) and cut the logs into manageable pieces, preparing them for further processing and storage. As mentioned earlier during the distributor stage, each ingester has four modes (technically five in older architectures of Loki if you include Pending, but since Handoff is deprecated, I will exclude it): JOINING: The ingester is currently inserting tokens into the ring and initializing itself. It may receive write requests for tokens it owns. ACTIVE: The ingester is fully initialized and may receive both write and read requests for tokens it owns. LEAVING: The ingester is shutting down and may receive read requests for data it still has in memory. UNHEALTHY: The ingester has failed to heartbeat and is set by the distributor during periodic ring checks. When a distributor sends a log stream to an ingester, the ingester builds the log stream into a set of "chunks" in memory and flushes them to the backing storage backend at a configurable interval, similar to processed logs being released to a warehouse. Chunks are compressed and marked as read-only when: The current chunk has reached capacity. Too much time has passed without the current chunk being updated. A flush occurs. When chunks are considered packed or read-only, a new writable chunk takes their place. If an ingester breaks down, all data that hasn’t been flushed will be lost. This is why replication is crucial for Loki's durability, and we usually recommend a minimum of 3 replicas. During a flush, logs are bundled up and given a unique barcode, known as a hash, which is a unique combination of tenant, labels, and contents (excluding the ingester hash due to replication). This prevents multiple ingesters with the same data from writing the same data to the backing store twice. If an ingester fails, multiple differing chunk objects may be created. We will discuss how we retrieve our logs later. For now, it’s safe to say the logs are being stored securely within the warehouse, also known as object storage. ==Nicole, do you want to take over and take us through the query components?== #### Query (Nicole) (Nicole) Sure, Jay! At this point, we've got our logs nicely cut up and waiting for us. Now it's time to actually use them. When we want to retrieve those logs, or at least parts of them, we do a type of structured search that we call a query. This part can be really difficult because there are so many logs to look through! In this section, we'll talk about three different components that all have to do with retrieving logs: - Query frontend - Query scheduler - Querier. It is worth noting that the Query frontend and scheduler are optional-- only the querier is required for Loki to function. But we'll assume all three of them are deployed in this example! #### Query frontend (Nicole) —-------> Query Frontend (Nicole) You can imagine the Query frontend like the order processing desk. Instead of a query going directly to the queriers, it goes to the query frontend. The query frontend: - holds all the queries in a queue until the queriers can work on them - splits large and complex orders (the queries) into many smaller ones so that multiple queriers can work on them simultaneously later. - When all the smaller orders have been worked on, the query frontend stitches them all back together. - The frontend also does some query caching. It remembers some of the most recent orders in case more of those orders come through. This is great for when a user makes repeated queries in quick succession, reducing the load on the queriers from needing to execute the same query repeatedly. #### Query scheduler (Nicole) —-------> Query Scheduler (Nicole) Once the query frontend, or order processing desk, has finished processing the orders, they're then sent to the query scheduler. The query scheduler is like a production scheduler that maintains the production queue. The schedulers are the one that assign jobs to queriers. Both the Query Frontend and Query Scheduler are stateless, but due to the in-memory queue, it’s recommended to run more than one replica to keep the benefit of high availability. Two replicas should suffice in most cases. #### Queriers (Nicole) —-------> Queriers (Nicole) Now for the workers on the query side: queriers! After the queries are put in the queue that the query scheduler maintains, the queriers come to the scheduler and request jobs. Once assigned, queriers actually execute Log Query Language (LogQL) queries. In our real-world example, you can imagine Queriers as cutting and shaping machines in the facility. - They process the log data to meet specific requirements, retrieving and shaping the pieces as needed to fulfil the orders (queries). Interestingly, when a querier is looking for logs to fill a job, it does not look in the object storage first. They query all ingesters for in-memory data before falling back to check the backend store. - In our real-world example, this means logs that match the cutting and shaping machine’s requirements can be taken directly from the production line of the debarking and sawing machines. If these production lines do not have the desired logs, the cutting and shaping machines will request the logs next from the warehouse. This greatly improves query performance by reducing the distance needed to retrieve logs. Earlier, we discussed what happens if an ingester fails and we end up with duplicate logs. This is where the querier’s next role kicks in by internally deduplicating data. It checks the log's timestamp to the nanosecond, label set, and log message-- and if all those are the same, then it deletes the exact copy. #### Index gateway (Jay) —-------> Break (Jay) We have now covered both the read and write path components of Loki. However, we are not quite done. Some components operate in the background and serve automation roles. Let's take a look! —-------> Index Gateway (Jay) Starting off, we have the index gateway, which you can imagine as the inventory management system. It keeps track of all the processed log pieces (indexed log data). It is responsible for handling and serving metadata queries. Metadata queries look up data from the index. The query frontend queries the index gateway for the log volume of queries to make a decision on how to shard the queries. The queriers query the index gateway for chunk references for a given query so they know which chunks to fetch and query. Index gateways can run in two modes: “simple,” meaning all index gateways are available for all tenants, and “ring,” which uses a similar mechanism to the ingesters' hash ring. ==I am going to hot potato you back to Nicole to explain the compactor!== #### Compactor (Nicole) (mime catching a hot potato and throwing it aside) —-------> Compactor (Nicole) Uh, thanks, Jay... I guess. The compactor in Grafana Loki can be compared to a system that organizes and compresses the inventory records of the facility, ensuring efficient storage and retrieval. What it's actually compacting are the index files. - It merges multiple index files produced by the ingesters into one at regular intervals. The compactor downloads the files from object storage, merges them into a single file, uploads the newly created index, and cleans up the old files. So it's doing a lot of tidying up. - The secondary role of the compactor is to perform log retention and deletion. We will cover this in more detail in another video. For now, imagine its role in clearing out logs that have passed their sell-by date or removing a batch of logs that a customer no longer needs from the warehouse. Next up is the ruler! #### Ruler (Jay) —-------> Ruler (Jay) The ruler is an interesting component. Its role is like the quality control department. It monitors the processed logs in two ways. The first is quality control, also known as alerts. We will expand on this in another video, but for now, know that alerting rules allow you to define alert conditions based on LogQL and send notifications about firing alerts to an external service. For instance, we might see a high intake of logs containing the log-level error. The second role is recording rules. Recording rules allow you to run intensive metric queries over your log data at given intervals. For example, they can give the current percentage of each log level over the last day. These metrics are then pushed into a time series database like Prometheus, so they can be retrieved by Grafana. This reduces the overall load on Loki for costly queries, as these long-range queries have pre-calculated results that can be called upon from an independent metrics store. Much like our log warehouse keeping a database of current inventory statistics. The ruler stores its rule list within object storage and is interacted with via an API. ==Let's jump back to Nicole again so she can teach you how Loki handles needle-in-a-haystack queries.== #### Bloom compactor and gateway (Nicole) —-------> Bloom Compactor + Gateway (Nicole) Gee, thanks for making ME talk about these, Jay. ;) Well, in this case it's less like a needle in a haystack and more like a log in a warehouse. See, occasionally we might need to find a specific log that's already in the warehouse, but we don't want to use any of the labels they've been sorted by. Maybe we want to look for something very specific in the log contents, like a traceID. You can imagine this would be difficult, and would probably take a lot of time. We'd have to look through every single log, one at a time...or do we? The Bloom Compactor is a Loki component that's like an advanced sorting and labelling system that builds detailed catalogues, known as blooms, for the processed logs stored in the facility. You can think of these blooms as storing the overall pattern of a log rather than its exact contents. When one of these difficult queries is made, another component, called the Bloom Gateway, goes through all the blooms that the Bloom Compactor has stored and compares them against the query. But instead of directly searching for what's in the query, the Bloom Gateway uses the patterns stored in blooms to eliminate whole sections of the warehouse so that it doesn't have to look through so many logs to find what we're looking for. It's a process of elimination that ends up being much faster. https://grafana.com/docs/loki/latest/operations/query-acceleration-blooms/ #### Outro (Jay) —-------> Outro (Jay) And I think we are there! As you can see, Loki is a complex machine with many components working together to provide efficient log aggregation and querying. Understanding these components and how they interact is crucial for deploying and maintaining a robust logging infrastructure. In our upcoming videos, we dive into helm deployments and their three deployment modes. #### Outro (Nicole) If you'd like to get started with Loki, check out this video, where Jay and I walk you through the steps to get Loki up and running on your local machine via Docker Compose. Happy logging! ## Related videos and resources 🎥 How to get started with Loki: https://www.youtube.com/watch?v=1uk8LtQqsZQ ## Pinned comment ``` ``` ## Timestamps 00:00 Intro 01:04 Distributors and hash rings 05:07 Ingesters 09:21 Query frontend 10:25 Query scheduler 10:56 Queriers 12:53 Index gateway 13:49 Compactor 14:39 Ruler ## Post-production - [ ] Create captions using Rev, then edit them. - [ ] Take a thumbnail photo and create one for YouTube (1280 x 720). - [ ] Add description. - [ ] Include the title and any keywords in the first few sentences, but in a natural way. - [ ] Add related videos and resources as links. - [ ] Add end screen. - [ ] Add cards if necessary. - [ ] Add chapters/timestamps for YouTube if longer than 5 minutes. - [ ] Add mid-roll ads/check monetization settings. - [ ] Add pinned comment. - [ ] Post video preview on Patreon. - [ ] Schedule on YouTube. - [ ] Add video to relevant playlist(s), or create a playlist if necessary. - [ ] Schedule on Mastodon. - [ ] Create a blog post on [site](https://nicolevanderhoeven.com). - [ ] Change `published` to `true`. - [ ] Change `publish` to `true`. - [ ] Change `status` to `Done`.