# [[Observability]] ![[Observability.svg]] Observability is the measure of how much can be inferred about the inner workings of an application based on its ouputs. [^simme] As a practice and a [[Tech|computing science]], observability involves setting up applications to produce outputs that can be studied, analyzing the outputs, and improving the application based on insights learned from the outputs. True observability involves gathering *more information about a system than you need*, so that you can answer not just expected questions but also *unexpected* ones. [^gl] ![[Introduction to observability#^2fc6d5]] ## [[Observability vs Monitoring]] Wait, what's the difference between observability and monitoring? In general, monitoring is a subset of observability in that monitoring is *one* way to make a system observable. For example, [[Continuous Profiling]] is another observability signal that may not fit into traditional monitoring. Observability is also a quality of the system, while monitoring is a specific activity that can facilitate the quality of being observable. ## Pillars of Observability - [[Metrics]]: alerting, dashboarding, AI/ML - [[Logs]]: due diligence, debugging, incident response - [[Traces]]: debugging, performance tuning - [[Synthetic monitoring]] - [[Load Testing]] - [[Continuous Profiling]] ## Levels of observability What part of the system you're observing affects the recommended approach, tools, and databases. - [[Frontend observability]] - [[Application observability]] - [[Infrastructure observability]] ## Tools and services for making a system observable - [[Observability Platforms]] are full, proprietary, end-to-end observability [[SaaS]] products that are one-stop shops for instrumenting and observing a system. - [[Observability Stacks]] are sets of open-source tools that are commonly used together to recreate end-to-end observability. - [[Observability Tools]] are [[Composability|composable]] utilities that focus on specific types of [[Telemetry]]. ## What should we observe? - [[RED Metrics]] - [[Four Golden Signals of SRE]] - [[USE Method]] ## [[Trends in Observability]] ## Concerns in observability - Cost - [[Alerting]] [^gl] [^simme]: Aronsson, S. (2021). _Intro to distributed tracing with Tempo, OpenTelemetry, and Grafana Cloud._ Retrieved from https://grafana.com/blog/2021/09/23/intro-to-distributed-tracing-with-tempo-opentelemetry-and-grafana-cloud . [[Intro to Distributed Tracing With Tempo, OpenTelemetry, and Grafana Cloud|My highlights]]. [^gl]: Grafana Labs. *Introduction to the pillars of observability*. Retrieved from https://university.grafana.com/learn/course/10/module-introduction-to-observability %% # Excalidraw Data ## Text Elements ## Embedded Files e7cd135f285f6c74c573547abab6b67cb9f07e30: [[icon-eye.png]] 015a722f4162d9a7b95bf9036cbf52e361e59f9e: [[icon-black-box.svg]] 60f66252dc985de60b7c2d0b4b24b9c20a14f2a7: [[Data Visualization.svg]] c7a64c0a7c0fea5747b208b2ac78434141f79d6d: [[Telemetry collector.svg]] ## Drawing ```compressed-json N4KAkARALgngDgUwgLgAQQQDwMYEMA2AlgCYBOuA7hADTgQBuCpAzoQPYB2KqATLZMzYBXUtiRoIACyhQ4zZAHoFAc0JRJQgEYA6bGwC2CgF7N6hbEcK4OCtptbErHALRY8RMpWdx8Q1TdIEfARcZgRmBShcZQUebQBGADYEmjoghH0EDihmbgBtcDBQMBKIEm4IAE0jACsAaShKgC0eUmIAM3bSUkSAJSEADUkASQBBVJLIWEQKwn1opH5SzG5n HkSAdgT4nniNngAWTfiAVgAODcSlyBhVk8SATm0zs5OeDfiAZhfPj6vCyAUEjqbifHgnZ47E4fd6bB48B4Ha5SBCEZTSUHgyHgmGXDbwxHI6zKYLcAAMyOYUFIbAA1ggAMJsfBsUgVanWZhwXCBbITUqaXDYWnKGlCDjEJkstkSDkcLk8rJQfmQdqEfD4ADKsFJEkEHhVECpNPpAHVgZJuHwAUbqXSENqYLr0PrysixeiOOFcmh4si2NzsGpbr6y RSbaLhHBhsQfag8gBdZFdXCZGPcDhCDXIwgSrAVXBkw1iiVe5hxzPZm1hBDEUEbMMHHifJLIxgsdhcNCfe5tpisTgAOU4Ym48TJ8M+iTJnwe/0mZWYABF0lBa9x2gQwsjNMIJQBRYKZbJxxPIoRwYi4Nd130bE4PB4nMlnU4tpE2ogcWkZrP4ZEssK65oJu+DbtWURQEIcYQIgEq5sohpqsE6YSAgGzYMQXwnO0PCvO0iTYBsBzYNCPYHBsuCCpo iQ0RhmgPO0ZIbAgnxFpS7jiPGAJgH6PHxACSY2tgNJwL+GqFAAvksxSlOUEgUAgADiFBCMQNSjMwTT6BQMAAGqSAACu08QABKEAAMoa0xcWU8zKIsNorGgziJIc2huZsPBkjwCLecRyIhqgzhsU8nzfO+bE4p8H4LkCxAgr6TYeW8iLxGcBwZcRRzIpIqLosqSVxIkqUHOlmUUQcOU2sSzrhguxr2lKrLsuQ8rcryyo7kKIolpKzItbKbUKp1SHq lqOq2a6daUnaZoWlas0mg6k0VNNxbCJ63pjv6gbBmOYbIpGF4xqeQkLimaa3qglb/jaubEPmEi4IkG3isQZYVn+lIIMBqB7BsgO+elZx9h2nBWnsYMDhww4cKOvoPPsFEfPsObLquf2geBC67u9h4ZEqZ3npe15/XsD5Psx8Jko8AG5j+aC3QBbBAdd2MILN17QRUcGOBwiHJuqCCoegZKnLg+w8O0ZVucQDySwxJyaO0DwzoRKtvKxiTxAgD6q4 5DWcfk/HXLxgnIiJAbifgUkyfd10QEuZK4PuhkAGIALJ6YZmqe8QmAkQAQrSpqaqZBzWfAtlzAshrOcFblktobxkpliK02ciQZYFqxJAc2jvD57wVfEZVI8i8WJf9yUlQiZUZVlVXzqUeVohiRUpfX5VN9VC61Vx9WlI19LNTK6ByiNSqGoKwrHRKY+tZyHXT0LGqOs6RrMm6EHLeaCWWmg1oNXNK1OlN28zTaHqSJ9O02gGQr7aGQ+QMd0axvk5 2lJdIvXcz908wJwgC9PSb1SzbSZt9asv1rrxEnCVMMU5oaditHhFBQ4RxcW8mXTKlxs7oxXMEG8G4tycxtHjA8R4iZfxJleEhd5KbPg2GcR8LZ6bfhtizNmpCwLkJPtzGCfMEJjRQo7cWJxJa+RlkkHg8tFZPhVmrKc2BNY8G1rrfWDxDbD2NmgAokxeJmwEpMb+kArZiSgRJEo0lCiyUgPJdAyhiBKWwAAVU1CcNgRg6gWSME0Qgwx3ZNDccweI bJkQ2VmPZHRkAE6uXiPEbQ+JEkvERNOCcLcbh5ziG8Mu04KK7EOGXDYlcFqdzrmlRulU+6t3yh3GuxVSo9xqVkkBAs6pLSagNce0Bhorz5N1OefVF5DWXoqQZNpkITXPmtS+hoR4IH3tXY+w9T4bwvgad0m1b6QP+rtJ+sADqvwgO/U6tCpnkCulwwBj1gEvXwOAj6eyAENVgWODKBxlFuVBjadsMMrQbE+Bg2GWDuAPDBDFZhbTCAY2IVjMhO49 zEAJseHIFyFwXnoeTe8j5mEXERKUz8DMbkLkAvSdmiKIKCN5lkfmgspnC1FhACRUjpayzkQrDYSslHq1Ue0LWU5NGMW0QsvR3FDF8UlRbYSokbZ2zsQ7ComBJBLlNFAJSpoViROjtEuOyIE7pW0FOD4ZUXyeRKgcYFNogprC+IXbyT5EihUOC+X5cVymoBikkmKQLfibBfEkNpbcCqgjKsaii4U8SBsSG0ge5Iumjx6UvdqEyuoUJ6vPfq0oU1T0 mRdcaGyKiSCFBoQICzT7LMPvs3e9oi16nmdsvwuzyz3wXI/IMRyX5Noga2qxd03l/SLucEF3A3Igrhgjf6bwWzrBOK2G0WKyZwNxVTe8U5akOLhQgBhqAOZHTFB/YmFDkWopofosxEAvyMxutAslrMKW8Jxj/K5f9SU/yZY7acBE3JvGINgB4rxHrTk0BhORZJNAHE0IcBi2BvK4DLrhSWYqCBcQMZMKVGGZULgse+hxQCCzjAVSUexZRHZNFwPC IOiRlDsRtFEiQscHLx1WJlOIJS5xknnVxj4PZc4uXBOxtOFEnxggbu60oVdq3PhOSGhpVVPgeRbDOUTJSXhEg6YPRNjJk1jNTaNIZvVkWjInv0tNoiZmbxLdgMtsTbR709ZhgQ6zVoNq2dfHZd9fQHM7UFccJy+pedvVWQdcCDgzliqUf5qDfTvAnWCpKb4IX+bocuscq78V4TOCc2FRCd0Ir4QeqM5yL1IvxtQk8GLSjXrw1eh9BXn2qlfaLV5H 6xEVCIi9EiLsMJknaCEaEFFoMvmg0KFhVqyplXaPiYgiQr5G1QybSVxjsOlFw/2nMBHnpBwgMRooSqJBCDcQcYY8QjAAHklxRxmGhfQmhaxXk0KSA1qx13aB8kcNipxY1gn48FMqWwezghOJlLjYI9iRcBJ6mTuV6mFVQDDmqmmE21qTTmiQABiXW2O7OzyM+9EzfTxkGcZevVz6BrO2YrQ5g+ba1nLXrS6RtHnm1BacxADtz9/qHRZ72r6IXh7v LQD8pso60A5z+f2Tsk6uLwPvEkBsbxUu7opni81pqcvbt3dSIQ/DShnM/qVk95XCaVaN2SklEg8h5AACrpEyNSGAqA9AagQNgKArJtCmGUAmBMhpyUNb1011Mb6NvVnFehkoTnzamMttBD3+glzXlwNwexkB9DEEMqJOMafIDQQQOd+wJAnArk3FmdFIEqWGMgHjrNntrw2YZNYegoQn1B4FJmvq9eoA2bPXyNAOv281878Z3T6AMedHaCqDvwzk Xnb2l21A04zY19ZI9Ug3fe8Vfh4PlfEBdxtCYITifnRp+r8P6Qefhygqvza1kEPelYWECe1xDmPFJIAlsQuB6T10C4AZLtjYvbAuGuJgPDhAIALwbgAszvW527UKO7O7MjBDu6e7e6+5IScBQCaiEBGDYInLtCYHuypjqhBSQ7QBYBQCjBEDKBdjoDBDtDpoLjthQDmAEBUFoi0HQDWw5iOySB6QABSWQAwmgYCOqN2E8FBLGR8JwimiQlUXGasg M2cchf2zgcuKcbwMhSCT4KMqyUOtOR8RwKcs6JwsO7c8O4IGmJIWmqOOm6O4+OOusM8I+BOY+RO+mq8pOlmtklOIgdmiyVadOzmDO5OW87mC4N8bOPmXOKWvOzyfawWA6guf0jwH2YuXq8W8MsuZc2WmwDYJyS6KuGW5q9w6whCmM10u+EYh6JW8Yl6lCKK2+x6FunCVutu9uO6pATuLuyBHupAXu9APufu3Cj6lehWlyIeLWd6a2mBuAuYTAUxA ukArIaIuYBAduYBFQ0BsBHRCBPRbufRAxQxxYlAGx4BOx8BXRiBruKB/RaBwxxKXopkcOY4hcphNUQgHuvQ4QOBXEVR3+W2f+V2Uk4A50ICcAcA2oZMqehQ0AeUmQFQ1BGISwDAhACAFAQcLhC8bhJ+k+U+KJ2AIgnUwwa4+g2oy0x+jhOiEAhJ3QSoJJGQmJs+rh9h7heajB5iRJ9JpJ7shaoR60BJXJ2QDJZJlanqqyNJQpUAIp5Jda/JzOa2U pIpvQnmeymGkpdJwppJV+vmxygpmp0pPJhBxB+ApB+pxJRp2Q2BuBVoQ8GpFpGQZx7BNBiJCADB/I9p3JGQUJpAlB3QbAFAeUuA/80xnpWpGQ+4Eoow/pgZIQjsPINIVA5pXp+g0ZiZNuuqEgfUHptJDp+g7sr6KpzorWRoVsGoAw3ArwSS+w9wTYCIPwXGKJzAZZ+AlQ3AJwMhxqYmmUzqhwHwKJRgbABgMJTBBAuu5I2gj4pwhwe2YZhpGQKp7 0QWEA2ZKJooJA1peBa5pAG5a4liCO25JAnsbAj0kZVEwQlK4x+uO52ag0qApGQczIjspAyggoAAFLsKUrwFDN+V+cnCcAAJSGjfHKBZg8gVAvnvnNgUi8BsTUCwUwX/lAWzm5lKiyn0jz4sGcD85JEQC/zfF5g7kCwjmlD34v6VGkC66WxED7n/GlAcAh7cB0XLGfFfh/GUVB4QDzCkD0ikCDiMUD4cXIjcW8VnnkVMUcWzl2A1AHHMCagMVwDHm nkMXiVjHPogLu6ECMA25DmPJoCkYMYujpCaUQxx5UgGAZniGtaQAB6XmNZ4U0hknGVYW0H7qfihCUGaXaW6Xyo2LgBf54XCw54f6SRAA ``` %%