%% Last Updated: - [[2021-02-11]] tags: %% Author: [[Simme Aronsson]] ## During my review - Start from the steady state - fix issues you already know about - Focus on "unknown unknowns" - Minimize blast radius to prevent cascading failures - [[chaos-toolkit]] from chaos-toolkit.org - [[Nginx]] pods via [[Kubernetes]] - `chaostoolkit-k6` and `chaostoolkit-kubernetes` extensions - 13:35 empty space - 13:51 repeated - He ran k6 through chaos toolkit before the experiment. chaos toolkit created a journal of the steady state. - Then he did the chaos experiment: terminate nginx pod. Hypothesis was still met. (checking was done using chaos toolkit python library). So he terminated two nginx pods. - 19:37-20:00 empty space ## Questions during the presentation Date: [[2021-02-11]] ### Can chaos testing be used for testing embedded devices? Yes, but not with the tools he used (chaos-toolkit). You can apply the concept to anything, but it might not be automated. ### Would you recommend building a full-fledged staging environment? What about test data? Yes, production-like is pretty important for chaos engineering. Start out in a staging environment and then test in production when you know what you're doing a bit more. Do backups before the test, but using real data has issues. Maybe you can just generate test data DURING the test. ### Chaos engineering for non-distributed/microservices? It might be difficult to have a monolith - one instance of a system - because you already know what would cause it to fail. But you still could introduce network latency or experiment with peripheral services, experiment on people systems, database systems. ### Is stress testing a form of chaos engineering? Yes! Any type of turbulence you're introducing into a system can be considered chaos engineering. ### Mocking? Mock third-party providers (if you can't send test flags to make sure they're not executed) but try to keep as much of the system intact as possible. ### Control kubernetes? k3s k9s cubectl ## See also - [[Chaos Engineering]] - [[Principles of chaos engineering]]