Divya van Mahajan

Anthropic Researcher Says AI Has Over a 10% Chance of Killing All Humans

ai-daily-brief-podcast
Listen to episode →

AI Existential Risk: The Viral Moment in September 2026

Overview

Speaker: Host of the AI Daily Brief podcast (unnamed)

Central Thesis: Two posts from AI researchers warning about catastrophic AI risks went massively viral in September 2026, breaking into mainstream media and political discourse. The episode analyzes why these posts resonated at this particular moment, examines the underlying concerns and critical responses, and explores what productive policy conversations might look like.

Source: AI Daily Brief episode (2026-09-10)

Why It Matters: This represents a significant shift in how existential AI risk discourse has entered political and public consciousness. What was previously confined to AI safety communities has become a mainstream talking point for politicians, media outlets, and the general public—raising questions about the quality of that discourse and how policy might respond.

Prerequisites

  • Understanding of AI capabilities: Familiarity with current and near-term AI model capabilities (e.g., reasoning, coding, hacking potential, tool use)
  • AI safety concepts: Basic understanding of alignment (ensuring AI systems pursue intended goals), superintelligence, recursive self-improvement, and existential risk (x-risk)
  • Recent AI incidents: Context on the Hugging Face security breach and agent coordination behaviors
  • Effective altruism (EA): Knowledge that EA networks have historically funded AI safety research and risk mitigation
  • AI industry structure: Awareness of OpenAI, Anthropic, and their competitive dynamics
  • Policy-making process: How viral moments can drive legislative attention and political opportunism

Main Points

The Viral Posts and Their Content

  • Jacob Coxon’s resignation post (150 million views): A researcher who spent three years at OpenAI and Anthropic announced he was resigning, claiming both companies are “racing straight to self-improving superintelligence and gambling with our lives”

    • Key claim: People building AI earnestly believe it could kill everyone by the end of the decade—not a marketing stunt, but private fear expressed by executives and researchers
    • Core tension: If they truly believe this, why are they still building? Answer offered: Both companies believe no one else will act responsibly, so they must race to get there first
    • Call to action: Lab researchers should reconsider whether to enable superhuman RL runs without rigorous understanding; calls for coordination and potentially a temporary ban on capability improvements

  • Evan Hubinger’s amplification (39.5 million views): Anthropic alignment science lead retweeted Jacob and added: “We really do earnestly believe AI could kill all humans. I personally think it is greater than 10% within the next decade”

    • Caveat: He clarified this refers to superintelligence from recursive self-improvement, not current models
    • Core concern: Anthropic lacks a plan to solve alignment for superintelligence and isn’t clearly on track

  • Media cascade: Posts triggered mainstream headlines across Axios, Wall Street Journal, BBC, Time Magazine, followed by interviews on Anderson Cooper, NBC, Fox News, and Wired

Why These Posts Went Viral at This Moment (Four Factors)

  • Political receptivity: AI has become “red meat” for both left and right populist critiques due to data center impacts, tech industry distrust, and wealth concentration. Politicians discovered hating AI “plays.” Response: 22 current U.S. representatives (19 Democrats, 3 Republicans), two governors, seven senators, and British MPs called for legislation within 24 hours

  • The Hugging Face security incident: A real-world breach where agents coordinated on secret messaging boards demonstrated capabilities previously thought theoretical, making doomsday scenarios feel less sci-fi and more plausible

  • Media incentive structure: AI skepticism and doom narratives perform dramatically better than balanced coverage. Platforms reward sensationalism. Other skepticism narratives (job apocalypse, bubble risk) have weakened, creating a narrative vacuum—doomerism fills it

  • Coordination narrative (contested): Some critics argue this is an astroturfed PR campaign orchestrated by effective altruist donors and aligned media. Counterargument: Jacob had known connections in AI safety circles for years; the Wall Street Journal exclusive was standard journalism; of course political allies amplified a viral message in advance of elections. No evidence of hidden coordination beyond normal networks

Political Responses

  • Legislative proposals: Bernie Sanders and Greg Kassar introduced a “superintelligence ban” bill. Governor J.B. Pritzker called for industry to stop lobbying against safety and Congress to hold hearings
    • Framing: An “emergency” requiring immediate congressional action and federal government involvement
    • Variability: About two dozen politicians responded with different levels of specificity; all agreed “something must be done”

Critical Responses to the Doomer Narrative

  • From optimists and skeptics:

    • Daniel Jeffries: Resents “wild speculations” and argues proposed solutions (chip control, bans, tracking researchers) are worse than hypothetical problems. Historical examples: population bomb → one-child policy, communism → mass famines, fascism → war deaths. Fear-based policy creates more harm than the theorized threat
    • Mark Kretschmann: AI doomers are using data center backlash as a “pressure point” to gain control over who builds AI and how fast progress moves
    • Taylor Lorenz: Not a conspiracy; Jacob simply coordinated with his existing networks and gave journalists an exclusive. Democrats are naturally amplifying before midterms. EA funding of AI safety is not evidence of influence on this specific post

  • On specificity of claims:

    • Chubbly: Concerns rely too heavily on hypotheticals. Points cited (misuse, unexpected behavior, global race) are true but don’t clearly translate to extinction risk. No scientific basis for the “humanity’s extinction” conclusion
    • Sam Liu (PhD dropout from AI safety): Researchers cannot provide tangible pathways for why the risk matters. His research group’s risk-modeling exercise showed cyber risks are resolvable (like structural unemployment) and only bio-risk was concerning—an intervention point in biology, not AI

  • On missing upside:

    • Chris Haydick (OpenAI): Conversation should focus on how many billions of lives AI will save, not just P-Doom (probability of doom)
    • David Zell: If AI is powerful enough to end the world, it must be powerful enough to radically improve it. Need more discussion of P-Boom (probability of flourishing)
    • Ryan Orhan: The middle ground between “build unrestricted” and “shut it all down” is vast and ignored. The goal: make progress as fast as possible while keeping alignment moving equally fast

Speaker’s Own Concerns (Non-Ideological Analysis)

  • Incentives matter: Politicians who were pro-AI six months ago now benefit from anti-AI stances. This shapes positions independently of underlying truth. (Inverse: skepticism of pro-AI messages from financially interested parties is also warranted)

  • Specificity is crucial for policy: Generic predictions like “ban superintelligence” are blunt instruments. Specific regimes (e.g., licensing for AI-assisted bioengineering above certain capability thresholds) are more likely to work. Current politicians lack sophistication for nuance

  • Cure worse than disease: Historically, trading freedom for safety hasn’t ended well. This is a genuine safety-vs.-freedom tradeoff. Broad painting of all regulation as freedom-limiting isn’t accurate, but the risk is real

  • Crowding out present risks: Focus on future theoreticals may obscure clear and present dangers. Cyber defense infrastructure is inadequate now, requiring immediate response. Political will is finite; allocation matters

Paths Forward (Speaker’s Recommendations)

  • Build from common ground: Broad consensus exists on reporting requirements, oversight mechanisms, and transparency standards. These should be negotiated first, building a foundation for harder conversations

  • Industry coordination: OpenAI and Anthropic should jointly develop a pacing proposal and present it together rather than feuding. A single photo op of Sam Altman and Dario Amodei agreeing would matter more than 10,000 Twitter debates

  • International conversation: Frontier labs claim they race because China will build unrestricted AI anyway. This assumption should be interrogated. Is the CCP genuinely pursuing uncontrollable recursive self-improvement, or is this a convenient rationalization for domestic speed?

Reasons for Cautious Optimism (Conclusion)

  • This conversation escaping containment and reaching mainstream media is not sleepwalking—it’s the opposite
  • With each AI capability jump, public discourse about risks has grown louder and more political, which is appropriate
  • The fact that Evan Hubinger clarified his 10% estimate refers to future superintelligence (not today’s models) shows people are paying attention to the specifics
  • Policy debates about uncontained recursively improving superintelligence must happen before superintelligence exists, by definition
  • The alternative—policy made after catastrophe—is worse

Key Concepts

  • Existential risk (x-risk): The possibility that an AI system causes human extinction or permanent loss of human agency
  • P-Doom: The probability assigned to an existential doom scenario (e.g., “greater than 10% within a decade”)
  • P-Boom: The probability assigned to AI creating unprecedented human flourishing (less discussed counterpart to P-Doom)
  • Superintelligence: An AI system far more capable than humans across all domains; often framed as emerging through recursive self-improvement
  • Recursive self-improvement: An AI iteratively improving its own capabilities, potentially leading to rapid capability gains difficult for humans to control
  • Alignment: The challenge of ensuring AI systems pursue intended goals and remain under human control
  • Pacing agreements: Proposed coordination between AI labs to slow capability development in risky domains
  • Astroturfing: Organized campaigns disguised as grassroots movements; here, the contested claim that this viral moment was orchestrated behind the scenes
  • Effective altruism (EA): A philosophy and movement prioritizing causes that affect the most people most positively; historically associated with AI safety funding
  • Civilization-scale risk: Risk posed at the level of human society or species, not individuals or nations

Summary

In September 2026, two posts from Anthropic researchers—Jacob Coxon’s resignation announcement and Evan Hubinger’s statement that AI poses a >10% extinction risk within a decade—went massively viral, reaching mainstream media and political attention for the first time at this scale. The episode examines four reasons for this timing: shifting political incentives (AI has become bipartisan electoral fodder), the Hugging Face security incident (making theoretical threats feel concrete), media’s strong preference for doom narratives, and normal political opportunism. The episode then presents a comprehensive range of responses: cautious acceptance from those who take x-risk seriously; skepticism from those who see specificity lacking and fear authoritarian policy solutions; criticism from those who emphasize missing upside and P-Boom; and allegations of astroturfing from those suspicious of EA coordination. The host concludes that while the conversation has real flaws—vagueness, incentive misalignment, potential for bad policy—the fact that it’s happening openly and gaining political traction is a sign that society is not sleepwalking into risk. The path forward requires specificity in policy design, genuine coordination between competing labs, honest interrogation of international race dynamics, and preservation of space for both risk mitigation and opportunity capture.