%% date:: [[2023-04-11]], [[2023-11-26]], [[2023-11-27]] parent:: [[Observability]] %% # [[Observability Tools]] Tools for [[Telemetry collector|collecting]] [[Telemetry]] data from application components. Unlike the end-to-end observability platforms, the tools below require some setup time and effort, but are [[Open source]] and can ultimately be more flexible. They are often thought of in terms of an "[[Observability Stacks]]" because they are separate projects that happen to work very well together. Recommended open-source setup: Prometheus + Grafana Loki + OpenTelemetry + Grafana ## Metrics - [[Prometheus]] - [site](https://prometheus.io/) - [[Graphite]] - [site](https://graphiteapp.org/) - [[statsd]] - [repo](https://github.com/statsd/statsd) - [[Nagios]] - [site](https://www.nagios.org/) - [[OP5 Monitor]] - [site](https://www.itrsgroup.com/products/network-monitoring-op5-monitor) - [[JMeter Perfmon Agent]] - [repo](https://github.com/undera/perfmon-agent) ## Logs ### Tailing and ingestion - [[Grafana Loki]] - [site](https://grafana.com/oss/loki/) - works well with Prometheus - [[Beats]] - [site](https://www.elastic.co/beats/) - [[Logstash]] - [site](https://www.elastic.co/logstash) - [[fluentd]] - [site](https://www.fluentd.org/) - [[Fluent Bit]] - [site](https://fluentbit.io/) - [[Loggly]] - [site](https://www.loggly.com/) ## [[Distributed tracing|Traces]] - [[Jaeger]] - [site](https://www.jaegertracing.io/) - [[Grafana Tempo]] - [site](https://grafana.com/oss/tempo/) - [[Zipkin]] - [site](https://zipkin.io) ### Instrumentation - [[OpenTelemetry]] - [site](https://opentelemetry.io/) = OpenTracing + OpenCensus - [[Grafana Agent]] ### Backends and visualization - [[Grafana Tempo]] - [site](https://grafana.com/oss/tempo/) and [repo](https://github.com/grafana/tempo) - highly available distributed tracing backend - [[Jaeger]] - [site](https://www.jaegertracing.io/) - [[Zipkin]] - [site](https://zipkin.io/) - [[Apache Skywalking]] - [site](https://skywalking.apache.org/) - [[Elastic APM]] - [site](https://www.elastic.co/apm) - Skywalking - Elastic APM ## Pre-processors These tools collect and/or transform the data in some way, whether from one format to another or to aggregate them, as an intermediary step before visualization or storage. - [[Kapacitor]] - [site](https://www.influxdata.com/time-series-platform/kapacitor/) - [[Telegraf]] - [site](https://www.influxdata.com/time-series-platform/telegraf/) ## Visualization - [[Kibana]] - [site](https://www.elastic.co/kibana) - [[Grafana]] - [site](https://grafana.com/) - [[Chronograf]] - [site](https://www.influxdata.com/time-series-platform/chronograf/) ## Data storage Monitoring data has to be stored somewhere. In recent years, [[Time-series database]]s have grown in popularity for this use case.