LLM Engineers Quitting Over AI Existential Risks: Concerns, Quotients, and Regulatory Responses (A Perplexity Report)
The Exodus of AI Safety Experts
In recent years, a growing number of artificial intelligence researchers and engineers have resigned from leading AI companies, citing concerns about the existential risks posed by increasingly powerful large language models (LLMs) and the race toward artificial general intelligence (AGI). This trend has been particularly notable at OpenAI, where nearly half of the company’s AI risk team has reportedly left.
Among the high-profile departures is Steven Adler, who worked as an AI safety researcher at OpenAI for four years before resigning in November 2024. Upon leaving, Adler expressed that he was “pretty terrified by the pace of AI development” and questioned whether humanity would even survive long enough for him to “raise a future family” or save for retirement. Adler specifically pointed to artificial general intelligence (AGI) as “a very risky gamble, with a huge downside” and emphasized that “no lab has a solution to AI alignment today”.
Other notable AI safety experts who have left OpenAI include:
• Miles Brundage, who led a team focused on AGI readiness policies
• Lilian Weng, who served as VP of research and safety
• Richard Ngo, an AI governance researcher who worked extensively on AI safety
• Ilya Sutskever and Jan Leike, who co-led OpenAI’s “superalignment” team
The departures extend beyond OpenAI. Dario Amodei, former Vice President of Research at OpenAI, left in 2020 to co-found Anthropic, citing concerns that OpenAI’s focus on scaling AI models was overshadowing necessary safety protocols. This pattern of resignations reflects growing anxiety within the AI research community about the trajectory of AI development and the potential risks it poses to humanity.
Understanding the “Extinction Quotient”
The term “extinction quotient” does not appear to be a standardized technical term in AI safety literature. However, it likely refers to the probability estimates that researchers assign to the risk of human extinction due to advanced AI systems.
Several surveys have attempted to quantify this risk:
1. A 2022 expert survey on AI progress found that approximately 50% of AI researchers believed there was at least a 10% chance that humans could go extinct due to their inability to control AI.
2. A more recent survey of 2,700 AI researchers found that almost 58% considered there to be at least a 5% chance of human extinction or other extremely bad AI-related outcomes.
3. Geoffrey Hinton, often called the “godfather of AI” and a Nobel Prize winner in Physics, has estimated a “10% to 20%” chance that AI could result in human extinction within the next three decades.
These probability estimates reflect the concept of existential risk from artificial intelligence—the idea that substantial progress in artificial general intelligence could potentially lead to human extinction or an irreversible global catastrophe. The core concern is that if AI were to surpass human intelligence and become superintelligent, it might become uncontrollable, similar to how the fate of other species now depends on human decisions.
The AI Alignment Problem
At the heart of these concerns is what researchers call the “AI alignment problem”—the challenge of ensuring that increasingly powerful AI systems remain aligned with human values and intentions. The alignment problem refers to the difficulty of building AI systems that reliably do what their operators want them to do, even as they become more capable and potentially autonomous.
As Steven Adler noted when leaving OpenAI, “No lab has a solution to AI alignment today. And the faster we race, the less likely that anyone finds one in time”. This sentiment reflects the technical challenge of ensuring that advanced AI systems:
1. Correctly interpret human instructions
2. Understand the full range of human values
3. Act in accordance with those values even as they become more capable
4. Avoid developing instrumental goals that could conflict with human welfare
The alignment problem becomes increasingly critical as AI systems approach or potentially surpass human-level intelligence. Without solving this problem, there’s concern that even well-intentioned AI systems could cause harm through misunderstanding human values or developing emergent behaviors that conflict with human welfare.
Proposed Legislation and Regulatory Approaches
In response to these concerns, various legislative and regulatory approaches have been proposed or implemented to mitigate the risks posed by advanced AI systems:
United States
1. The Preserving American Dominance in AI Act: Introduced by a bipartisan team of Senate lawmakers in December 2024, this legislation would establish the Artificial Intelligence Safety Review Office to evaluate and test frontier AI models for extreme risks before deployment.
2. Advanced AI Security Readiness Act: Introduced in June 2025, this bipartisan legislation directs the National Security Agency (NSA) to develop an AI security playbook to defend American advanced technology against foreign adversaries.
3. Executive Orders: The regulatory landscape in the US has seen shifts with changing administrations. In January 2025, a new executive order titled “Removing Barriers to American Leadership in AI” replaced previous regulations, prioritizing AI innovation and U.S. competitiveness.
European Union
The EU has taken a more comprehensive approach with the EU AI Act, which was formally adopted in mid-2024 and became effective on August 1, 2024 (though most provisions won’t be enforced until August 2, 2026). The EU AI Act:
1. Takes a risk-based approach, categorizing AI systems based on their potential harm
2. Prohibits certain “unacceptable risk” AI applications, including:
3. Cognitive behavioral manipulation
4. Social scoring systems
5. Biometric identification in public spaces (with limited exceptions)
6. Imposes strict requirements on “high-risk” AI systems, including those used in critical infrastructure, education, employment, and law enforcement
Global Initiatives
There has been an increase in AI-related standards activities and cross-jurisdictional collaboration, exemplified by initiatives driven by organizations such as:
1. The Organisation for Economic Co-operation and Development (OECD)
2. The US National Institute of Standards and Technology (NIST)
3. UNESCO
4. The International Organization for Standardization (ISO)
5. The Group of Seven (G7)
Additionally, 2024 saw an increased focus on AI safety with the launch of new AI safety institutes in the US, UK, Singapore, and Japan.
Can the Existential Threat Be Eliminated?
The question of whether the existential threat posed by advanced AI can be completely eliminated remains contentious among experts. Several perspectives emerge from the research:
Arguments for Possibility of Risk Mitigation
1. Technical Solutions: Researchers at institutions like the Alignment Research Center, the Machine Intelligence Research Institute, and the Center for Human-Compatible AI are actively working on technical approaches to the alignment problem.
2. Regulatory Frameworks: Comprehensive regulatory frameworks like the EU AI Act aim to prevent the development of the most dangerous AI applications while ensuring safety measures for high-risk systems.
3. International Cooperation: Some legal scholars argue that there is an international legal obligation on states to mitigate the threat of human extinction posed by AI, grounded in the precautionary principle and the right to life within international human rights law.
Arguments for Persistent Risk
1. Competitive Pressures: As Steven Adler noted, “Even if a lab truly wants to develop AGI responsibly, others can still cut corners to catch up, maybe disastrously. And this pushes all to speed up”. This creates a dangerous race dynamic that makes safety harder to ensure.
2. Technical Challenges: The alignment problem remains unsolved, and some researchers argue that it may be fundamentally difficult to ensure that superintelligent systems remain aligned with human values.
3. Inherent Uncertainty: The nature of advanced AI systems, particularly those approaching or exceeding human intelligence, makes their behavior inherently difficult to predict with certainty.