EA - New 80,000 Hours problem profile - Artificial intelligence by Benjamin Hilton

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: New 80,000 Hours problem profile - Artificial intelligence, published by Benjamin Hilton on August 29, 2022 on The Effective Altruism Forum. We (80,000 Hours) have just released our longest and most in-depth problem profile — on reducing existential risks from AI. Some background Like much of our content, this profile is aimed at an audience that has probably spent some time on the 80,000 Hours website, but is otherwise unfamiliar with EA -- so it's pretty introductory. That said, we hope the profile will also be useful and clarifying for members of the EA community. The profile primarily represents my (Benjamin Hilton's) views, though it was edited by Arden Koehler (our website director) and reviewed by Howie Lempel (our CEO), who both broadly agree with the takeaways. I've tried to do a few things with this profile to make it as useful as possible for people new to the issue: I focus on what I see as the biggest issue: risks of power-seeking AI from strategically aware planning systems with advanced capabilities, as set out by Joe Carlsmith. I try to make things feel more concrete, and have released a whole separate article on what an AI-caused catastrophe could actually look like. (This owes a lot to Carlsmith's report, as well as Christiano's What failure looks like and Bostrom's Superintelligence.) I give (again, what I see as) important background information, such as the results of surveys of ML experts on AI risk, an overview of recent advances in AI and scaling laws I try to honestly explain the strongest reasons why the argument I present might be wrong I include a long FAQ of common objections to working on AI risk to which I think there are strong responses Also, there's a feedback form if you want to give feedback and prefer that to posting publicly. This post includes the summary from the article and a table of contents. Summary We expect that there will be substantial progress in AI in the next few decades, potentially even to the point where machines come to outperform humans in many, if not all, tasks. This could have enormous benefits, helping to solve currently intractable global problems, but could also pose severe risks. These risks could arise accidentally (for example, if we don’t find technical solutions to concerns about the safety of AI systems), or deliberately (for example, if AI systems worsen geopolitical conflict). We think more work needs to be done to reduce these risks. Some of these risks from advanced AI could be existential — meaning they could cause human extinction, or an equally permanent and severe disempowerment of humanity.2 There have not yet been any satisfying answers to concerns — discussed below — about how this rapidly approaching, transformative technology can be safely developed and integrated into our society. Finding answers to these concerns is very neglected, and may well be tractable. We estimate that there are around 300 people worldwide working directly on this.3 As a result, the possibility of AI-related catastrophe may be the world’s most pressing problem — and the best thing to work on for those who are well-placed to contribute. Promising options for working on this problem include technical research on how to create safe AI systems, strategy research into the particular risks AI might pose, and policy research into ways in which companies and governments could mitigate these risks. If worthwhile policies are developed, we’ll need people to put them in place and implement them. There are also many opportunities to have a big impact in a variety of complementary roles, such as operations management, journalism, earning to give, and more — some of which we list below. Our overall view Recommended - highest priority This is among the most pressing problems to work on. Scale AI will have a variety of impacts a...