# [[kOH 61 - Resilience and chaos testing with SteadyBit and k6]] <iframe width="560" height="315" src="https://www.youtube.com/embed/GhyKF7oP21M" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> ## Post-production - [ ] Add this to my vault somewhere. - [ ] Add this to nicolevanderhoeven.com. - [ ] Add this to my newsletter. ## Dev log - Followed [instructions here](https://docs.steadybit.com/quick-start/deploy-example-application) to deploy the demo application. - [[SteadyBit shopping demo]] ### [Install agents](https://docs.steadybit.com/quick-start/set-up-agents) - Log onto [the SteadyBit platform](https://platform.steadybit.io) and follow the instructions to install agents for Kubernetes ([[Helm for Kubernetes]]). - Run this command for [[Docker]]: (This was what was in the docs, but with the `cluster.name` set to `steadybit-demo`) ```bash helm repo add steadybit https://steadybit.github.io/helm-charts helm repo update helm install steadybit-agent --namespace steadybit-agent \ --create-namespace \ --set agent.key=b68aiy0bd7nhto \ --set cluster.name=steadybit-demo \ --set agent.registerUrl=https://platform.steadybit.io \ --set agent.containerRuntime=docker \ steadybit/steadybit-agent ``` The result was: ```shell nic@sopirulino shopping-demo % helm repo add steadybit https://steadybit.github.io/helm-charts helm repo update helm install steadybit-agent --namespace steadybit-agent \ --create-namespace \ --set agent.key=b68aiy0bd7nhto \ --set cluster.name=steadybit-demo \ --set agent.registerUrl=https://platform.steadybit.io \ --set agent.containerRuntime=docker \ steadybit/steadybit-agent "steadybit" already exists with the same configuration, skipping Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "steadybit" chart repository ...Successfully got an update from the "grafana" chart repository ...Successfully got an update from the "prometheus-community" chart repository ...Successfully got an update from the "stable" chart repository Update Complete. ⎈Happy Helming!⎈ NAME: steadybit-agent LAST DEPLOYED: Wed Aug 31 15:43:59 2022 NAMESPACE: steadybit-agent STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: It may take a few moments for the agents to fully deploy. You can see what agents are running by listing resources in the steadybit-agent namespace: kubectl get all -n steadybit-agent You can get the logs for all of the agents with `kubectl logs`: kubectl logs -l app.kubernetes.io/name=steadybit-agent -n steadybit-agent -c steadybit-agent nic@sopirulino shopping-demo % ``` Wait a few minutes for the agent to appear on the SteadyBit platform. It took ages, so I went digging: ```shell nic@sopirulino shopping-demo % kubectl get all -n steadybit-agent NAME READY STATUS RESTARTS AGE pod/steadybit-agent-hvksw 0/1 ImagePullBackOff 0 24m NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/steadybit-agent 1 1 0 1 0 <none> 24m nic@sopirulino shopping-demo % kubectl logs -l app.kubernetes.io/name=steadybit-agent -n steadybit-agent -c steadybit-agent Error from server (BadRequest): container "steadybit-agent" in pod "steadybit-agent-hvksw" is waiting to start: trying and failing to pull image nic@sopirulino shopping-demo % ``` Ugh, I hate Docker. Opened up Docker and reran the helm command. Didn't work! ``` Error: INSTALLATION FAILED: cannot re-use a name that is still in use ``` Honestly, I just waited and put this aside. The next day, I could click *Continue* on SteadyBit and the agents had connected! ![[steadybit-agents-connected.png]] Ok, created an attack (container stop) on SteadyBit UI. ``` nic@sopirulino shopping-demo % kubectl get pods -n steadybit-demo NAME READY STATUS RESTARTS AGE fashion-bestseller-689779dbfb-f7pph 1/1 Running 0 19h gateway-76d65f8558-rxgp4 1/1 Running 2 (12h ago) 19h hot-deals-6b9789777f-54tgd 1/1 Running 0 19h toys-bestseller-5d76c4fdcc-cx8q9 1/1 Running 0 19h ``` Tried to run experiment, but now it's saying there are no active agents. Ok, deleted namespace with `kubectl delete namespace steadybit-demo` and did a `kubectl delete -f`. Redoing the demo app. ``` nic@sopirulino shopping-demo % kubectl get svc -n steadybit-demo NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE fashion-bestseller NodePort 10.99.12.28 <none> 8082:32272/TCP 2m2s gateway LoadBalancer 10.104.71.51 10.104.71.51 80:31605/TCP 2m2s hot-deals NodePort 10.104.135.236 <none> 8083:31749/TCP 2m1s toys-bestseller NodePort 10.107.141.247 <none> 8081:30705/TCP 2m1s ``` Working well, k6 script works too. Tried to install agents again, realised that the namespace I deleted was `steadybit-demo` not `steadybit-agent` , which is what the error was about. Deleted `steadybit-agent` namespace and ran again. Finally working! ``` nic@sopirulino shopping-demo % kubectl delete namespace steadybit-agent namespace "steadybit-agent" deleted nic@sopirulino shopping-demo % helm repo add steadybit https://steadybit.github.io/helm-charts helm repo update helm install steadybit-agent --namespace steadybit-agent \ --create-namespace \ --set agent.key=b68aiy0bd7nhto \ --set cluster.name=steadybit-demo \ --set agent.registerUrl=https://platform.steadybit.io \ --set agent.containerRuntime=docker \ steadybit/steadybit-agent "steadybit" already exists with the same configuration, skipping Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "steadybit" chart repository ...Successfully got an update from the "grafana" chart repository ...Successfully got an update from the "prometheus-community" chart repository ...Successfully got an update from the "stable" chart repository Update Complete. ⎈Happy Helming!⎈ NAME: steadybit-agent LAST DEPLOYED: Thu Sep 1 10:35:11 2022 NAMESPACE: steadybit-agent STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: It may take a few moments for the agents to fully deploy. You can see what agents are running by listing resources in the steadybit-agent namespace: kubectl get all -n steadybit-agent You can get the logs for all of the agents with `kubectl logs`: kubectl logs -l app.kubernetes.io/name=steadybit-agent -n steadybit-agent -c steadybit-agent nic@sopirulino shopping-demo % ``` Agent is running. ``` nic@sopirulino shopping-demo % kubectl get pods -n steadybit-demo NAME READY STATUS RESTARTS AGE fashion-bestseller-689779dbfb-kcbjj 1/1 Running 0 48m gateway-76d65f8558-fh4v6 1/1 Running 3 (45m ago) 48m hot-deals-6b9789777f-2mjnj 1/1 Running 2 (46m ago) 48m toys-bestseller-5d76c4fdcc-jp2nz 1/1 Running 0 48m ``` Run experiment. This one is meant to stop the `fashion` pod: ``` nic@sopirulino shopping-demo % kubectl get pods -n steadybit-demo NAME READY STATUS RESTARTS AGE fashion-bestseller-689779dbfb-kcbjj 0/1 Error 0 49m gateway-76d65f8558-fh4v6 1/1 Running 3 (46m ago) 49m hot-deals-6b9789777f-2mjnj 1/1 Running 2 (47m ago) 49m toys-bestseller-5d76c4fdcc-jp2nz 1/1 Running 0 49m nic@sopirulino shopping-demo % kubectl get pods -n steadybit-demo NAME READY STATUS RESTARTS AGE fashion-bestseller-689779dbfb-kcbjj 0/1 Running 1 (8s ago) 49m gateway-76d65f8558-fh4v6 1/1 Running 3 (46m ago) 49m hot-deals-6b9789777f-2mjnj 1/1 Running 2 (47m ago) 49m toys-bestseller-5d76c4fdcc-jp2nz 1/1 Running 0 49m ``` Looks like that worked. Now to integrate with [SteadyBit API](https://docs.steadybit.com/integrate-with-steadybit/api): Token: `BRc5dFEI.vRRB2neaTWClNYh5qnGflGvUg6p09RAL` ### Documentation issues - In [Deploy on Minikube](https://docs.steadybit.com/quick-start/deploy-example-application#step-2a-deploy-on-minikube) > Deploy the example application, the command should be `kubectl apply -f k8s-manifest.yml` - The postgres container is no longer used. The resulting pods are: ``` nic@sopirulino shopping-demo % kubectl get pods --namespace steadybit-demo NAME READY STATUS RESTARTS AGE fashion-bestseller-689779dbfb-f7pph 1/1 Running 0 2m19s gateway-76d65f8558-rxgp4 1/1 Running 0 2m19s hot-deals-6b9789777f-54tgd 1/1 Running 0 2m18s toys-bestseller-5d76c4fdcc-cx8q9 1/1 Running 0 2m19s ```