# [[Instrumentation]] ![[Instrumentation.svg]] Instrumentation is an approach to [[Monitoring]] that involves modifying either the code or the binary of the thing to be monitored, with the goal of exposing information about it and making it more [[Observability|observable]]. ## Types of instrumentation ### By method of instrumentation #### [[Source instrumentation]] Source instrumentation is the process of adding code that saves the information you require during runtime, in a specific format. [[SDK]]s, libraries, and other tools such as [[OpenTelemetry]] facilitate source instrumentation. [[Grafana Faro]] is a type of source instrumentation for the frontend. #### [[Binary instrumentation]] Binary instrumentation involves modifying the binary of the application/infrastructure component without modifying the underlying code. Binary instrumentation could involve installing an [[Telemetry collector]] on components you want to monitor. The agent collects the information and then communicates metrics to the rest of the observability stack. ![[Instrumentation diagram.png]] ### External instrumentation [^grcevski] There are some approaches to instrumentation that, strictly speaking, do not directly instrument either the source or the binary. We can group these into "external". The [[JVM|Java Virtual Machine]] lets you extend it with additional agents with appropriate permissions to observe some signals from it. Another form of external instrumentation involves listening for the information based on the protocol, such as what [[eBPF]] tools do. The exact approach for eBPF varies, but they don't fall into source or binary either: - In Java, you can use eBPF to tap into the Linux kernel and monitor signals, then pair those signals together. - In Go, you can attach to the binary (sort of like a debugger, without modifying/instrumenting the binary itself) and listen for signals at breakpoints or established "safe" periods within the program. ### By intervention required Instrumentation can be automatic or manual: - [[Auto-instrumentation]] uses standards for frameworks, languages, or protocols (such as [[OpenTelemetry]] or [[Grafana Alloy]]) to automatically detect information about the component and then insert code (at the source or binary level) to expose the information. - [[Manual instrumentation]] involves modifying the source or binary explicitly to detect and expose information. [^grcevski]: Grčevski, N. (2024). [[GOH 25 - Introduction to eBPF with Grafana Beyla, with Nikola Grcevski]] %% # Excalidraw Data ## Text Elements ## Embedded Files c3a6e19fe578edc275ce2ed951bbd0189b1d7df6: [[icon-server.svg]] a9237a51729e503817bfd98df81877614c5aa5c5: [[icon-bug-red.svg]] f3f0b5478da54a90fe80e67b54bbcd4823ae44f9: [[icon-signal-transmit.svg]] {"fill":"#ff0000"} ## Drawing ```compressed-json N4KAkARALgngDgUwgLgAQQQDwMYEMA2AlgCYBOuA7hADTgQBuCpAzoQPYB2KqATLZMzYBXUtiRoIACyhQ4zZAHoFAc0JRJQgEYA6bGwC2CgF7N6hbEcK4OCtptbErHALRY8RMpWdx8Q1TdIEfARcZgRmBShcZQUebQBGADYEmjoghH0EDihmbgBtcDBQMBKIEm4ICgApAGVSI3xMAC1UkshYRArCfWikflLMbmckgAZtAA5EgFZxgBZZgHYeacSe Kf7IGCH4xe0AZhGdnh54+IBOHZHVjcqSdW4TuMTxpKmZ8b2F8bOrm8kEQjKaQPeJPF7Td6fb6/QqQazKYLcEY3ZhQUhsADWCAAwmx8GxSBU0dZmHBcIFsq1SppcNgMcp0UIOMRcfjCRJiRxSeSslAqZAAGaEfD4GqwRESSS0jSBfkQVHorEAdTukm48RRaMxCDFMAl6EEHjljKBHHCuTQGthEDYZOwai2lpGyOtjOZZuYFtQHCEIpRCAQxAeiUS8 xujBY7C4aEm4aYrE4ADlOGJuHseAtZjwzrMpnxrUI4MRcFBA+qFlMzjmzt89p8zjdCMwACLpUtBtACghhG4M4RwACSxC9eQAujdNMJmQBRYKZbIj8fWogcDHcH1+5dsOllzvdhA3AXkTJD9e+/CH4UIU8SbB7XCJBDnAUIKZfQPYDNTMQ8QNnKbxJomjEIc3yaPExALMQAqJHKzDuOIqAFG0YBWih8SwkubQQNg6JwGem7YYQzJYBUuAjHBURQEI XoQIgzLEcoECFAAvv0xSlOUEiJtiVQCgAaswUyzHKHSIWUPTKH01qDGgJzaEs5xzJM6YvGsNyOqgzhTGMqnPP+FbnFMexoaUFCquqiRjPMyzjAsNbjDwan5th/yAsClpWdoNnPPZ4yOc5NzwvqYw5nsbxnBm3xTFZzzrNaCraqyBJEuQXJkhSfITrS9JuiyeIpRyaXcplcpCiKur6lI0oiNJ2GJcqFmWpqio6uK4mGuUNwmpIHpeqZkC2rSDrqs6 3VTsQfUEReCUBh2qB7L5CxxpGnAPHsDbWhGCYcMmHCppa3xLItak3IWxbtuWlbVhcGbXNaTatsEl17vgPbWn2hZDouE4TbOGS8j9y7EWuaAbjN2H4ju81dm9B7WkeuAnvN4OXsEN7oLgkWfLgAFLGcr4jHsLwLJoArEDW0EvHZCyJDs2BTLguMM3BCH5LCqEbKhmE3LhtrTY2JGyZj8SUSWNEVPRjgcExrHsQ982VPo048PoVQLAAEgAigASpIZy aAAsjrzAAAqG3sGsa6J8Did0vRysL8QLHs2jHBcdmHM78R2S5pSac4PAjHEyw8Hs8yLUTAGJDc5nEPcclJAkVwViMsz+U5oLjH8AJAnyifJPEKc6enAVZ0FMshS1SUFey6CciVvJyjSdKfcyyV19AxUZU3aOiu1FSdUG1eNfHapySPbV6h1eJda6wimua6o3EN9qwKNLrYZ9g7DvkWGlIjyMCw9QtkTwxoTVNYPnv6u6oKcwezDWYZbfGUbqrMMe v6tu0pohQczE/uMN4X8iItjbHfWG71sKTiZMQf684cjH2wudEsd9nbXR0jwRYzs9g3BXKDb0N8tzQ24FA+G9UqISwkFLRiZUrwYwgAKPYAoRiaGEl8YswksYjBfOMEYCBEik2EkBbAxB05h1wAgeYAoziswIIhZCbQBrczaPvSAfN8LXxFKxcAWEIC4DgHAMUaDuAcWgP8TIFQiB5ypAwQgCAKAACEcpt3ymyCoABiAUPjfF2OwCITKA5Sz6DFK1 DuXj4hPmif4wJvJgkZBca3PKESiokh7pSfoOE4nZASfoAAYsKfu09B6z2HoUbJpAgkhLCdqFUY8HhZICVU+JNStRYkqjPI0TSclQDyTrBevUl7NQqc06pGQADydoRpOk3ho3peT8mcCgPkpGwpNKzB6S03JISlnZBqIQIw/85mVPGfoAAKlgKAABBGx0Z0DBAFFlUZCy2nkhuVUtgFB/i4BRsQ0oYzWkZGnMya5nzvkhEVuSdEVAtlnLBTC85tsK h5VidsvpuzjwIAGfqVGFT4LohFAADW4AsU42gdIbSuOpfFfMRQAE1uCzBGFMCYjkdinHiKpUE8VShGDYAYMxW0CBCEQhhEobEXnov6ZfYZ6BUVZIZCQA5RyHgnKVcQMUCAtGoF5ZADVhs2DEAQCC3Amhggw33Iq0gJBUmoA4pAJxeJFakGUDSAAFCcZavBnbUB9d6sYUwACUcodYIGUL6ckKK3W4E9QcP1YdkS8HjagQNIa5ZSsyrUrEUyoBRi9H ig+WKw0kRtTLIV2EshmotdwNEoreZEB1XWihpQOBI0Qs2leQgoArg7aQet1oeikCxKQRM7ba39pbZAIdI7TXmrvs2jNpQ7AACsEDYByDUNtcBDXGrnTW160C4QbsIIwc5Ar8AVtKGJQe6QT1rV5jRKABgkWdG0RDUoUMsSWrhoedEoS715rWoeqdEB8ChBuSes9F7pq6LAJKwUV4vTABYiAFiQA= ``` %%