Alex Soto 57 m $ slot machine win blamed on software error, casino won't pay jackpot triggered even though he didn't have five of a kind - testing embedded systems is hard - defensive programming model - log hardware events to be able to double check - don't accept compensation Tesla model s safety solution - problem with auto pilot - struck rubber of tire and the whole car burned - solution: software update from home -- very disruptive lessons - testing embedded systems is hard - failure is a given - just need to detect them - break the rules: - mutation test: test the tests. change variables and make tests fail Windows 3.1 killed a french airport - weather reporting to plane - lessons: decisions now affect later - IT updates should have a plan B - hybrid/multi cloud - advanced deployment techniques blue green deployment - blue is old, green is new - keep the blue deployment running so that you can just reroute to it if you need to sf train - bart train ran off the track - 1973 - testing in prod is important - canary releases - what starts wrong, ends badly the fall of knight capital - high frequency trading using AI - $440M - "dark pool" is like NYSE but private - power peg for testing purpose (for extreme cases like buying when high, selling when low - exact opposite) - they releases 1st of aug when everyone is on PTO - last replica had flag set to true. $440M in 45 min - lessons - dark launches: setting headers? - do static analysis when deploying -- just check! - monitor business metrica - use gitops Power crisis in NA - 5 days without electricity - due to race condition - lessons: do stress testing, static code analysis, 3k inmates released early (60-600 days early) - WA sentence reduction algorithm changed - this happened for two years - lessons: maths are hard, do property-based testing: create matrix of values -