# [[Principles of chaos engineering]] ## Get the prerequisities right - An environment (production or test, or both) - Monitoring or [[Observability]] tools - Tools for notifications and alerts (like [[VictorOps]] and [[PagerDuty]]). - A company culture that encourages chaos ## Focus on measurable output Instead of internal attributes of a system (like resource utilization, queue depth, etc), build a hypothesis around metrics that the system can be measured by from the outside, such as error rates, latency, and uptime. ## Focus on critical systems, not low-hanging fruit. - Source: [[Chaos Engineering with Tammy Butow]] - It can be tempting to address issues that are non-critical, but easily fixed. This might be a good approach for functional issues, but chaos engineering is less exhaustive than functional testing. Chaos experiments provide value best when they target critical components that could potentially take the application down. ## Simulate real-world events Aim to replicate events that occur in the real world, even if you think they're unlikely to occur for your application. ## Have regular [[Chaos Days]] - Chaos days are like [[Periodic Game Days]] in performance and reliability testing. They are sometimes called Failure Fridays, and emphasize that chaos experiments should be run regularly to encourage a culture of chaos in a company, where developers are forced to write more resilient code. ## [[Minimize the blast radius|Start with a small blast radius]] - When beginning with chaos engineering, it's best to start with smaller experiments that affect one or few components, and then increase the "blast radius" or areas affected. Doing the reverse, starting with a large-scale experiment that affects the whole chain, may lead to too many failures to track down. - Minimizing the blast radius prevents cascading failures. ^57c243 ## Resources - [[sources/Presentation/Using Chaos Engineering to Test Distributed Systems]] - [[sources/Article/Using Chaos Engineering to Test Distributed Systems]] ## See also - [[Chaos Engineering]]