%% date:: [[2022-07-18]], [[2022-09-20]], [[2023-11-30]], [[2024-07-10]], [[2024-07-10]], [[2024-12-06]] parent:: [[Observability]] Siblings:: [[Metrics]], [[Traces]] %% # [[Logs]] ![[Logs.svg]] Logs are a pillar of [[Observability]] and are more verbose than [[Metrics]]. Logs help in diagnosing *why* problems occurred. [^horovits] Logs are chronological records of events and other output while a component or service is running. When compared to [[Metrics]] or [[Traces]], logs are usually the first type of [[Telemetry]] that is created. That's because developers often output logs to the console while troubleshooting their own code. The ubiquity of logs makes them the most familiar type of signal to work with. Logs are particularly useful because with the right parsing and aggregation, you can turn a log line into a metric or trace. [^bekker] ## Issues with collecting logs - *Difficulty in parsing*: Logs don't have a standard format or structure, and they could even be saved as different file types (CSV, JSON) - *Storage*: When left unchecked, verbose logs can present a problem in terms of storage, which forces us to answer the question of how long we want to keep what logs for. - *Ephemeral logs in ephemeral pods*: We need to find a way to save logs even when they're created from temporary containers like those in [[Kubernetes]] clusters. - *Aggregation*: Distributed computing means distributed logs, and it can be difficult to collect logs from many different components. - *Querying*: Logs need to be collected and organised in a format that will allow them to be easily retrieved later. ## Tools for collecting logs - [[Grafana Loki]] - [[Lucine]] - [[Logstash]] (for ingesting) + [[Elasticsearch]] (for storage and querying) - [[InfluxDB]] - [[TimescaleDB]] - [[Apache Druid]] - [[OpenTSDB]] - [[ClickHouse]] - [[Prometheus]] can be used with custom exporters to collect logs, even though it's focused on [[Metrics]]. It can also be integrated with Loki or [[fluentd]] - [[fluentd]] - [[Victoria Metrics]] - [[QuestDB]] - [[CrateDB]] - [[Cassandra]] (+ [[KairosDB]]): not a [[Time-series database|TSDB]], but stores logs %% # Text Elements # Drawing ```json { "type": "excalidraw", "version": 2, "source": "https://github.com/zsviczian/obsidian-excalidraw-plugin/releases/tag/2.1.4", "elements": [ { "id": "4y8R7iOA", "type": "text", "x": 118.49495565891266, "y": -333.44393157958984, "width": 3.8599853515625, "height": 24, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "roundness": null, "seed": 967149026, "version": 2, "versionNonce": 939059582, "isDeleted": true, "boundElements": null, "updated": 1713723615080, "link": null, "locked": false, "text": "", "rawText": "", "fontSize": 20, "fontFamily": 4, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "", "lineHeight": 1.2 } ], "appState": { "theme": "dark", "viewBackgroundColor": "#ffffff", "currentItemStrokeColor": "#1e1e1e", "currentItemBackgroundColor": "transparent", "currentItemFillStyle": "solid", "currentItemStrokeWidth": 2, "currentItemStrokeStyle": "solid", "currentItemRoughness": 1, "currentItemOpacity": 100, "currentItemFontFamily": 4, "currentItemFontSize": 20, "currentItemTextAlign": "left", "currentItemStartArrowhead": null, "currentItemEndArrowhead": "arrow", "scrollX": 583.2388916015625, "scrollY": 573.6323852539062, "zoom": { "value": 1 }, "currentItemRoundness": "round", "gridSize": null, "gridColor": { "Bold": "#C9C9C9FF", "Regular": "#EDEDEDFF" }, "currentStrokeOptions": null, "previousGridSize": null, "frameRendering": { "enabled": true, "clip": true, "name": true, "outline": true } }, "files": {} } ``` [^horovits]: Horovits, D. (2022). *5 key observability trends for 2022*. InfoWorld. Retrieved from: https://www.infoworld.com/article/3648053/5-key-observability-trends-for-2022.html [^bekker]: Bekker, W. (2023). *Getting started wit hGrafana Loki (Grafana Office Hours #09*. Retrieved from: https://www.youtube.com/watch?v=OLebNPLIJMI