<iframe width="560" height="315" src="https://www.youtube.com/embed/bmroQsJxTL8" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> ## Thumbnail ## Title ## What value do viewers get? ## Hook (Image of simple application.) (schema in excalibur?) To monitor an application like this, we could instrument every component, collect data with the Grafana Agent, send the data to Prometheus, and then visualize in Grafana. But manually instrumenting everything quickly gets out of hand in a modern application that looks more like this (show image of a microservices-based app with lots of components). You have to do it for each component-- even if they're using the same protocol. What if instead of instrumenting components, we could just expose metrics for standard protocols automatically? What if we didn't have to do any instrumentation for the common protocols at all? ## Intro Hi, I'm Daniel González Lopes, a Site Reliability Engineer at k6. I'm Artem Charykov, a Site Reliability Engineer? at k6. And I'm Nicole van der Hoeven, a Developer Advocate for k6. For this hackathon, we built a tool we're calling Teapot that collects networking data (TCP and HTTP) on the fly and exposes it as metrics and logs. But here's the bottom line: after installing Teapot, you get instant observability anywhere: no instrumentation required. ## Demos ### Daniel ### Artem ## The explanation So how did we do it? Well, we leveraged an interesting new technology called eBPF. eBPF stands for extended Berkeley Packet Filter, but that's not a great description of what it does. It allows developers to run programs in an operating system kernel without having to change kernel source code or use kernel modules. What that means is that instead of having a node exporter at this level (image of something at a high level), we're running a program at this level (zoom in to something analogous to a kernel). That program is watching for and responding to low-level networking-related events, regardless of the service that triggers them. It turns out that monitoring at the kernel level comes with some advantages. 1. It's highly efficient and performant, with an overhead that is conservatively estimated to be at 2%. 2. It works on the fly with any codebase regardless of language, even for external services that you may not want to instrument. ## The market You may be wondering: who else is experimenting with eBPF? You might recognize some of these names. Datadog includes eBPF monitoring in their Datadog agent. It supports capturing low-level networking information. Datadog’s network monitoring product uses it. A startup called Polar Signals has developed a lightweight agent called Parca-agent. It generates CPU profiles from running code using eBPF. Pixie, by New Relic, is a tool to instantly troubleshoot applications on Kubernetes. It supports a wide range of data sources and works like magic. But using it without their “managed” offering is hard, there is no good way to expose Prometheus metrics, and it's quite complex to operate. Here's what Pixie looks like (image of Pixie's subcomponents). What we want is something light and composable, like this (cross out the ones we don't want). Also, big companies like Netflix, Facebook, and Cloudflare use eBPF auto-instrumentation internally for all kinds of different purposes. ## The future Where do we go from here? First, we want to support all of the most common protocols. But after that, eBPF opens up other avenues beyond just networking. It could be used to improve security, create netflow charts, do capacity planning, and monitor SLIs out of the box. Imagine if we could integrate Teapot into Grafana Agent. Then, all we'd need to do would be to install Grafana Agent onto, for example, a Kubernetes cluster, and we'd get instant, no-instrumentation observability for the entire stack. We could have ready-made dashboards in Grafana Cloud. Teapot could even integrate with Grafana Machine Learning. Suddenly we have a wealth of data and rich insights about our application, and all of it by just installing a single, small binary called Teapot. Thanks for listening, and check out our work at the hackathon-2021-12-teapot repo! ## Related videos and resources https://docs.google.com/document/d/13cA4t6FskX6YPtX4FYO0I8DGlWOjHl3Hdg26Tz3_s2Y/edit# ## Pre-production - [ ] Take a thumbnail photo and create one for YouTube (1280 x 720). - [ ] Add description. - [ ] Include the title and any keywords in the first few sentences, but in a natural way. - [ ] Add related videos and resources as links. - [ ] Add end screen. - [ ] Add cards if necessary. - [ ] Add video to relevant playlist(s), or create a playlist if necessary. - [ ] Create captions using Descript or using the script, then edit them. - [ ] (For k6 videos) Send Floor thumbnail and video link for promotion. - [ ] Add chapters for YouTube if longer than 5 minutes. - [ ] Schedule on YouTube. ## Post-production - [ ] Promote on personal social media. - [ ] Twitter - [ ] LinkedIn - [ ] Create a blog post on [nicolevdh.com](http://nicolevdh.com) - [ ] Include in a weekly newsletter. ## Timestamps 0:00 Intro 0:50 What is Teapot? + demo 2:23 How it works: eBPF 3:17 The market for eBPF 4:03 The future of Teapot