- Author: [[80000hours.org]] - Full Title: Why Bill Gates and Others Are Concerned About AI, and What to Do About It - URL: https://80000hours.org/problem-profiles/positively-shaping-artificial-intelligence/ - ### Highlights first synced by [[Readwise]] [[2020-09-16]] - Rapid progress in machine learning has raised the prospect that algorithms will one day be able to do most or all of the mental tasks currently performed by humans. This could ultimately lead to machines that are much better at these tasks than humans. - Humanity’s superior intelligence is pretty much the sole reason that it is the dominant species on the planet. If machines surpass humans in intelligence, then just as the fate of gorillas currently depends on the actions of humans, the fate of humanity may come to depend more on the actions of machines than our own. - In October 2015 an AI system named AlphaGo shocked the world by defeating a professional at the ancient Chinese board game of Go for the first time. A mere five months later, a second shock followed: AlphaGo had bested one of the world’s top Go professionals, winning 4 matches out of 5. Seven months later, the same program had further improved, crushing the world’s top players in a 60-win streak. In the span of a year, AI had advanced from being too weak to win a single match against the worst human professionals, to being impossible for even the best players in the world to defeat. This was shocking because Go is considered far harder for a machine to play than Chess. The number of possible moves in Go is vast, so it’s not possible to work out the best move through “brute force”. Rather, the game requires strategic intuition. Some experts thought it would take at least a decade for Go to be conquered. - The advances above became possible due to progress in an AI technique called “deep learning”. In the past, we had to give computers detailed instructions for every task. Today, we have programs that teach themselves how to achieve a goal - One survey of the 100 most-cited living computer science researchers, of whom 29 responded, found that more than half thought there was a greater than 50% chance of “high-level machine intelligence” – one that can carry out most human professions at least as well as a typical human – being created by 2050, and a greater than 10% chance of it happening by 2024 (see figure below). - When superintelligent AI arrives, it could have huge positive and negative impacts - Here is a highly simplified example of the concern: The owners of a pharmaceutical company use machine learning algorithms to rapidly generate and evaluate new organic compounds. As the algorithms improve in capability, it becomes increasingly impractical to keep humans involved in the algorithms’ work – and the humans’ ideas are usually worse anyway. As a result, the system is granted more and more autonomy in designing and running experiments on new compounds. Eventually the algorithms are assigned the goal of “reducing the incidence of cancer,” and offer up a compound that initial tests show is highly effective at preventing cancer. Several years pass, and the drug comes into universal usage as a cancer preventative… …until one day, years down the line, a molecular clock embedded in the compound causes it to produce a potent toxin that suddenly kills anyone with trace amounts of the substance in their bodies. It turns out the algorithm had found that the compound that was most effective at driving cancer rates to 0 was one that killed humans before they could grow old enough to develop cancer. The system also predicted that its drug would only achieve this goal if it were widely used, so it combined the toxin with a helpful drug that would incentivize the drug’s widespread adoption. - And all it takes is for a single super-intelligent machine in the world to receive a poor instruction, and it could pose a large risk. - The smarter a system, the harder it becomes for humans to exercise meaningful oversight. And, as in the scenario above, an intelligent machine will often want to keep humans in the dark, if obscuring its actions reduces the risk that humans will interfere with it achieving its assigned goal. You might think ‘why can’t we just turn it off?’, but of course an intelligent system will give every indication of doing exactly what we want, until it is certain we won’t be able to turn it off. An intelligent machine may ‘know’ that what it is doing is not what humans intended it to do, but that is simply not relevant. Just as a heat-seeking missile follows hot objects, by design a machine intelligence will do exactly, and literally, what we initially program it to do. Unfortunately, intelligence doesn’t necessarily mean it shares our goals. As a result it can easily become monomaniacal in pursuit of a supremely stupid goal. The solution is to figure out how to ensure that the instructions we give to a machine intelligence really capture what we want it to do, without any such unintended outcomes. This is called a solution to the ‘control’ or ‘value alignment’ problem. - how to align it with human values. - **Note**: What if instead of trying to teach AI human values, we start with a human and turn it into a machine? The original [[Satina]] could be the basis for such a machine, with successive implementations and iterations of Satina. - Researchers in this field mostly work in academia and technology companies such as Google Deepmind or OpenAI. You might be a good fit if you would be capable of completing a PhD at a top 10 program in computer science or a similar quantitative course (though it’s not necessary to have such a background). We discuss this path in detail here: Career review of technical AI research - How do we avoid an ‘arms race’ in which countries or organizations race to develop strong machine intelligences, for strategic advantage, as occurred with nuclear weapons? - What are the key organizations you could work for? We keep a list of every organization that we know is working on AI safety, with links to their vacancies pages, here. The most significant organizations, all of which would be good places to work, are probably the following: Google DeepMind is probably the largest and most advanced research group developing general machine intelligence. It includes a number of staff working on safety and ethics issues specifically. See current vacancies and subscribe to get notified of new job openings. Google Brain is another deep learning research project at Google. See current vacancies and subscribe to get notified of new job openings. The Future of Humanity Institute at Oxford University was founded by Prof Nick Bostrom, author of Superintelligence. It has a number of academic staff conducting both technical and strategic research. See current vacancies and subscribe to get notified of new job openings. OpenAI was founded in 2015 with the goal of conducting research into how to make AI safe and freely sharing the information. It has received $1 billion in funding commitments from the technology community. See current vacancies and subscribe to get notified of new job openings. The Machine Intelligence Research Institute (MIRI) was one of the first groups to become concerned about the risks from machine intelligence in the early 2000s, and has published a number of papers on safety issues and how to resolve them. See current vacancies and subscribe to get notified of new job openings. The Cambridge Centre for the Study of Existential Risk and Leverhulme Centre for the Study for the Future of Intelligence at Cambridge University house academics studying both technical and strategic questions related to AI safety. See current vacancies and subscribe to get notified of new job openings. The Berkeley Center for Human-Compatible Artificial Intelligence is very new, but intends to conduct primarily technical research, with a budget of several million dollars a year. See current vacancies and subscribe to get notified of new job openings. The Future of Life Institute at MIT does a combination of communications and grant-making to organizations in the AI safety space, in addition to work on the risks from nuclear war and pandemics. See current vacancies and subscribe to get notified of new job openings. Alan Dafoe’s research group at Yale University is conducting research on the ‘global politics of AI’, including its effects on international conflict. PhD or research assistant positions may be available. AI Impacts is a non-profit which works on forecasting progress in machine intelligence and predicting its likely impacts. The Center for a New American Security is a think tank in Washington D.C. that has a program called the Artificial Intelligence and Global Security Initiative. Their research agenda is largely focused on long-term issues. - Learn more The top two introductory sources are: Prof Bostrom’s TED talk outlining the problem and what can be done about it. The Artificial Intelligence Revolution, by Tim Urban at Wait But Why. (And also see this response). After that: The key work describing the problem in detail is Superintelligence. If you’re not ready for a full book try this somewhat more technical article by Michael Cohen. If you want to do technical research our AI safety syllabus lists a range of sources for technical information on the problem. It’s ideal to work through these articles over a period of time.