%% date:: [[2024-01-16]] parent:: %% # [[What is Continuous Profiling and What is Pyroscope - with Ryan Perry]]  by [[Henrik Rexed]] (with [[Ryan Perry]]) ## About continuous profiling - Continuous profiling is one of the critical signals (along with logs, metrics, traces, and events) - Problem with continuous profiling - It was generating so much overhead - We had to start and stop the recording of continuous profiling manually - It's also not always straightforward to generate the right action in the environment to look at the right profile - Then, in 2010: Google-wide profiling paper talked about continuous profiling in production - What's been done to solve these problems? - Continuous profiling doesn't record everything - it samples at an interval of 10 seconds - it doesn't look at all components - Now overhead is at 1-10% - No need to generate right action, because it's run in production - Continuous profiling can also be run within CI/CD processes and then profiles can be compared - How to use continuous profiling - instrument code by adding a profiler - profiling storage-- like [[Grafana Pyroscope|Pyroscope]] - profiling visualization tool -- like [[Grafana]] - [[Brendan Gregg]] introduced [[Flamegraphs]] as the way to visualize continuous profiles - Instrumentation - Different languages have different profilers to add in the code - [[Go]]: pprof - [[Ruby]]: rbspy - [[Python]]: py-spy - [[Rust]]: pprof-rs - [[C#]]: dotnet trace - [[PHP]]: phpspy - [[eBPF]]: kernel code, could reduce overhead further - In continuous profiling, you can also create tags and attach them to a profile. You can attach a method or function to the profile so you can find it easier. - Tools: [[Parca]] and [[Grafana Pyroscope|Pyroscope]] ## About Pyroscope - Go, Java, PHP, NodeJS, Ruby, Python, Rust, and eBPF - What makes it unique - can ingest profiles from Pyroscope agents but also from any other agent on the market - Adhoc profiling: you can upload profiles to Pyroscope, store it there, and also visualize it - You can export metrics out of the Pyroscope agent - will generate Prometheus metrics - Server (to display profiles) + agent to collect it - Installation - Kubernetes: has a helm chart - Deployed as a [[DaemonSet]] - Two kinds of agents - attached to the library of the application or process of the application (like Java, .NET, PHP) - instrument code itself (like Go) - [[FlameQL]] is like [[PromQL]] but for profiles. - query language - lets us filter profiles based on tags - UI: tag view, single view, comparison view - Pyroscope [[OpenTelemetry]] integration - only available for Go, Java, Ruby - Once Pyroscope agent is open, you can link Pyroscope to the trace provider in our code-- this links the trace to the profile for more context - You use the Pyroscope OpenTelemetry library to attach the OTel SDK to the Pyroscope library. This will add extra tags to the trace for profiles, as well as extra Pyroscope attributes - Tracing exemplars currently available on Grafana Cloud but will be available in OS version - By default, Pyroscope agents are configured to push metrics to Pyroscope server. But you can enable the polling mode to pull instead. ## Interview with [[Ryan Perry]] - co-founder of Pyroscope (with Dmitri) - They were using rubyspy and really loved the information they got out of it. - Name: *Pyro* from flamegraphs + *scope* to be able to zoom in - New features in the future? - Add Pyroscope to your testing suites, testing and CI/CD pipelines - Cloud product: tracing exemplars, but this is coming to opensource version - more and deeper integrations - improvements to eBPF agent - tags for testing? - Yes - you should be able to get a high-level profile but also be able to break it out per test, branch (PR vs main), and also see profiles in tests over time - Extend to more languages - You can upload as well as retrieve profiles via API endpoint - When can we expect the first release of OTel profiling? - Ryan is on the working group -