%% date:: [[2023-11-18]] parent:: %% # [[Learning from failure with Elon Musk]] speaker:: [[Steve Upton]] conference:: [[TestCon Europe 2023]]  - Arianespace Ariane 5 rocket failed spectacularly because of a single line of code that hadn't been sufficiently system tested. - Usually, rockets are two-stage: the bits that are burnt up for rocket fuel are ejected and never used again. SpaceX has pioneered the science of recovering that so that it's not as expensive, but the attempt is a huge technical challenge in itself. - Plans have to be coherent: Based on our understanding of reality, it is possible. It is consistent with reality. - Example of fast feedback: SpaceX was parachuting the rocket back to earth) to try to do this, but it didn't work. So they stopped doing the thing that didn't work: *DAMPING STRATEGY*. - Instead, they pivoted to another idea: what if they could use the same rockets to take off to land? This is also a coherent idea. - This took A LOT of experimentation, but finally they did it in December 2015. They were able to land the rocket on land, and then later on a drone ship at sea. - *AMPLIFICATION STRATEGY*: They could see progress, so they did more of it. They added things, threw more engineering effort towards it, until it worked. - Even when they were experimenting and "failing", SpaceX was still actually making money from the launches because the first stage was still successfully going on missions. They were just experimenting on the second stage, the part that would have burnt up anyway. The experiments were *safe-to-fail*. - Things about the experiments that SpaceX got right - ✅ Coherent - ✅ [[Safe-to-fail]] - ✅ Fast feedback - ✅ Damping strategy - ✅ Amplification strategy - Elon did not apply these principles to Twitter/X - Blue checkmarks existed to verify that people are who they say they are. Instead, he made it available to anyone who would pay $8/month. - ❌ Coherent - ✅ Fast feedback - Lots of people told him that the plan wasn't working. - Twitter lost 50 of top 100 advertisers - Clear and unambiguous - BUT, Elon decided to ignore it and blame woke leftists - ❌ Safe-to-fail - Companies that had been spoofed actually saw their stock market valuations decline as a result - Lasting financial and reputational damage to Twitter - ❌ Damping strategy - Elon did not relent - ❌ Amplification strategy - Elon didn't do "more of what worked"; he removed the parts that did work. - [[Hypothesis-Driven Development]] - We believe that *this capability* - will result in *this outcome*. - We will know when we have succeeded when *we see a measurable signal*. (Here's how we're going to test the hypothesis.) - [[Thinking in Bets]], Annie Duke - We learn from failure more than from success. - We often just pat ourselves on the back when we succeed. - When we fail, we need to fill in the gaps of our understanding and have to rethink our views of the world. - The failure of a coherent experiment that was safe-to-fail is not really a failure. It's a learning. > We need some form of agreement on what pathways have *utility* and which we should explore, and which are dead ends. - [[Dave Snowden]] How do we test for coherence? - If you can make a Breaking Bad meme about your idea, it's probably not a good idea. - "Has this been tried before?" - "Does our understanding of reality allow this?" - Go to the [[First principles]]. - Do the laws of physics allow this? - "Can we get there from here?" Giving good feedback - [[Compliment Sandwich]] - [[Ritual Dissent]] - Someone proposes an idea. - The proposer puts on a mask or turns their back to indicate a ritual separation between the idea and the character of the proposer. - The rest of the group tear the idea (not the person of the proposer) apart. They're not allowed to say anything positive. - The idea is refined and improved. - This is the complete opposite of the compliment sandwich. Another mistake of Elon Musk's: Twitter is the code. - Twitter is the people. Twitter is the connections and the social network. - Problem when you treat a people problem like an engineering problem: people remember. - You can always build a new rocket. You can't make people forget. - When you violate the trust of your users, people remember. Teams, organizations, and users remember. - This is why [[Safe-to-fail]] is important: you can't `git reset` an organization. How do we make our experiments safe-to-fail? - Ritual dissent - ~~"Does this idea sound good?~~" "How will this fail?" - How could this go wrong? - Shorten [[Feedback]] cycles. - You should try weird things, but not just because they're weird.