The Last Checkpoint
On artificial intelligence, autonomous weapons, and the dissolution of accountability
"Some uses are simply outside the bounds of what today's technology can safely and reliably do."
Prologue The Problem of the Waiting Weapon
Somewhere in the cold dark of the North Atlantic, decades ago, a weapon waited. It had no impatience. It had no doubt. It simply listened — and when it heard what it believed to be an enemy, it was designed to act.
The CAPTOR mine — an encapsulated torpedo moored to the seafloor — represented one of the earliest serious experiments in autonomous lethality. The concept was elegant in its simplicity and terrifying in its implications. A passive sonar system would classify an acoustic contact. If the classification exceeded a threshold, the weapon would release. No human being would make the final call. No human being would be there to hesitate.
Engineers working on advanced torpedo systems in that era understood viscerally what the concept entailed. The ocean is not a clean acoustic environment. Thermal layers bend sound. Biologics — whales, shrimp, fish — generate interference. Neutral merchant vessels transit the same waters as enemy submarines. A weapon lying dormant for weeks or months has no awareness that a ceasefire may have been declared, that a diplomatic cable has changed everything, that the contact it is about to classify has already been positively identified as friendly by an asset the weapon cannot see.
The engineers worried. They were right to worry. The battle-space they were designing for was epistemically humble — it knew what it did not know. The question was whether the weapons systems they built would share that humility. The answer, then as now, was not obviously yes.
Part I The Bayesian Problem: What the Machine Cannot Admit
To understand why the current confrontation between Anthropic and the United States government matters beyond its immediate headlines, one must first understand something about how intelligent systems fail — and how the best of them are designed to fail gracefully rather than catastrophically.
In the world of acoustic signal processing and weapons fire control, engineers in the 1980s and 1990s grappled seriously with the mathematics of uncertainty. Standard Bayesian networks offered a principled framework: assign prior probabilities to hypotheses, update them as evidence arrives, output a posterior distribution. The system knows it is uncertain. The uncertainty is visible and manageable.
More sophisticated practitioners turned to Dempster-Shafer evidence theory — a framework specifically designed to handle what philosophers call the "known unknown." Where Bayesian methods require that probabilities sum to one across a defined hypothesis space, Dempster-Shafer allows a decision system to explicitly park unresolved belief in a state of acknowledged ignorance. It can say: "I have evidence. The evidence is insufficient. I do not know." That honest output — structured ignorance — is precisely what a human decision-maker needs to exercise sound judgment about whether to fire.
Technical Context
Dempster-Shafer theory handles what Bayesian networks cannot: genuine epistemic ignorance. Rather than forcing probability mass to distribute across known hypotheses, it permits explicit representation of "unknown" as a separate category. This is essential in acoustic classification where contacts may fall entirely outside the training library — novel signatures, environmental distortion, adversarial spoofing. The combination rule also requires that evidence sources be independent, a requirement that is extraordinarily difficult to satisfy in practice and whose violation produces dangerously overconfident fused assessments.
The critical limitation of both frameworks — acknowledged by practitioners who used them honestly — was the "unknown unknown." You can represent ignorance about which known hypothesis is correct. You cannot represent ignorance about possibilities that have not entered your hypothesis space at all. The novel submarine class. The acoustic signature deliberately engineered to evade your classifier. The situation that is simply unlike anything in your library. This is represented in the movie "The Hunt for Red October" where the automatic classified forced the new propulsion system into the "seismic anomaly" class.
What Dempster-Shafer could do was flag the boundaries of its own competence. A well-implemented system would not silently misclassify a contact outside its training distribution. It would raise its hand — in the language of mathematics — and say that something was happening that it could not confidently explain.
Large language models — the technology at the center of the current political crisis — cannot do this. They have no principled mechanism for representing genuine ignorance. When they encounter inputs outside their training distribution, they do not flag uncertainty. They generate confident-sounding text. The system has no internal experience of doubt to communicate.
This is not a minor technical caveat. In the context of weapons systems, it is potentially catastrophic.
Part II From the Ocean Floor to the Pentagon: A Contract Becomes a Crisis
In July 2025, Anthropic — the San Francisco AI company founded by former OpenAI researchers in 2021 — signed a contract with the United States Department of Defense worth up to $200 million. It was a significant moment. Anthropic became the first frontier AI company to deploy its models in classified military networks, working through defense contractor Palantir. The company's AI model, Claude, was embedded across intelligence community workflows and armed services operations.
Anthropic had built its identity around a specific claim: that it was the responsible actor in a landscape of reckless ones. Its founders had left OpenAI precisely over concerns about safety. Its internal documents articulated ethical constraints — a "constitution" governing what Claude would and would not do. When it signed the Pentagon contract, it believed it had secured assurances that Claude would not be used for two specific applications: mass domestic surveillance of American citizens, and fully autonomous lethal weapons systems.
Then came Venezuela.
On January 3, 2026, the United States military conducted an operation against Venezuelan President Nicolás Maduro. Details remain classified. But through its partnership with Palantir, Anthropic came to suspect that Claude had been used in connection with that operation in ways that may have crossed the lines the company believed it had drawn. An Anthropic employee raised internal concerns. The fragility of the contractual protections suddenly became apparent.
"New language framed as compromise was paired with legalese that would allow those safeguards to be disregarded at will."
— Anthropic statement, February 26, 2026
Weeks of negotiation followed. The Pentagon's position was clear: it wanted the right to use AI models it contracted for "all lawful purposes," without carve-outs determined by the companies that built them. The Department of Defense's Chief Technology Officer Emil Michael stated publicly that the military would "not commit in writing to any company that it will limit its own defensive capabilities." Publishing usage restrictions, the Pentagon argued, amounted to publishing the limits of American military AI capability — an intelligence gift to adversaries.
Anthropic's position was equally clear and rooted in two distinct concerns. First, a technical argument: current large language models are simply not reliable enough for autonomous lethal applications. They hallucinate. They generate confident wrong answers. Deploying them without human oversight in life-and-death decision chains "would endanger America's warfighters and civilians." Second, a constitutional argument: mass domestic surveillance of Americans violates fundamental rights — a position supported, Anthropic noted, by the Supreme Court's 2014 unanimous ruling in Riley v. California, which established robust Fourth Amendment protections against warrantless government access to personal digital data.
Part III The Ultimatum
On February 24, 2026, Defense Secretary Pete Hegseth summoned Anthropic CEO Dario Amodei to the Pentagon. The meeting was not a negotiation. It was an ultimatum. Relent by 5:01 p.m. on Friday, February 27 — allow unrestricted use of Claude for all lawful military purposes — or face consequences that would threaten the company's existence.
Hegseth threatened to invoke the Defense Production Act, a wartime statute that grants the federal government sweeping authority to compel domestic industries to prioritize national defense requirements. He threatened to designate Anthropic a "supply chain risk" — a label previously reserved for companies doing business with foreign adversaries like China, a designation that would require military contractors to cease using Anthropic products in their defense work. The practical effect could have been devastating: eight of the ten largest American companies use Claude, and the prospect of those companies being forced to choose between Anthropic and Pentagon contracts represented an existential commercial threat.
Amodei did not relent.
"I believe deeply in the existential importance of using AI to defend the United States and other democracies, and to defeat our autocratic adversaries. However, in a narrow set of cases, we believe AI can undermine, rather than defend, democratic values."
— Dario Amodei, CEO of Anthropic, February 26, 2026
On Friday afternoon, February 27, President Donald Trump posted on Truth Social. Every federal agency was directed to immediately cease using Anthropic's technology. Hegseth designated Anthropic a supply chain risk — the first time in American history that designation had been applied to a domestic company. Hegseth told military contractors that no company doing business with the Department of War could conduct commercial activity with Anthropic. Trump called Anthropic "left-wing nut jobs." The Under Secretary of War for Research and Engineering posted on X that Amodei was "a liar" with "a God-complex."
Hours later, rival OpenAI announced it had reached a deal with the Pentagon. OpenAI CEO Sam Altman had privately supported Anthropic's position — he had told employees the previous evening that his own company shared Anthropic's red lines — but OpenAI's approach differed in mechanism. Rather than hard contractual prohibitions, OpenAI accepted the "all lawful purposes" framework but secured architectural protections: cloud-only deployment, a proprietary safety stack the Pentagon agreed not to override, and cleared OpenAI engineers embedded in operations. The Pentagon accepted this. It had not accepted Anthropic's version of essentially the same principles.
The asymmetry was striking and unexplained. Both companies drew the same lines. One was banned. One was welcomed. The difference, as far as observers could determine, was not about the substance of the red lines but about who held contractual control over enforcement.
Part IV The Legal Battle and Its Implications
Anthropic immediately announced it would challenge the supply chain risk designation in federal court. The company cited 10 U.S.C. § 3252, arguing that the statute authorizing supply chain risk designations applies only to the use of Claude in Department of War contracts — it cannot extend to commercial customers, individual API users, or enterprise contracts serving non-military clients.
"We believe this designation would both be legally unsound and set a dangerous precedent for any American company that negotiates with the government."
— Anthropic statement, February 27, 2026
Legal scholars immediately questioned the government's procedural footing. The supply chain risk designation statute requires, among other things, that the Pentagon demonstrate it has exhausted less intrusive alternative measures before making such a finding. Given that the Anthropic dispute escalated from ultimatum to designation within days, experts questioned whether the government could credibly claim to have made a good-faith effort to find alternative solutions. Cornell Law professor Michael C. Dorf argued publicly that both mass surveillance and autonomous weapons deployment may already be unlawful — under the Fourth Amendment and customary international law respectively — and that the Pentagon's unwillingness to accept contractual carve-outs for clearly illegal activities raised an alarming inference: that the administration intended to pursue precisely those activities.
The business stakes were immediately understood. One independent analyst framed the dilemma precisely: even if Anthropic ultimately prevails in court, it could take years to resolve. And in the interval, every general counsel at every Fortune 500 company with Pentagon exposure would ask the same question — is using Claude worth the legal uncertainty? That question alone was capable of doing commercial damage that a court victory could not undo.
Anthropic's planned IPO, reportedly targeting a valuation of $380 billion, was placed in immediate uncertainty. The company's enterprise ecosystem — estimated at $14 billion, far exceeding the $200 million Pentagon contract — faced potential disruption as corporate customers assessed their exposure.
Part V The Battle Already Lost: Surveillance, AI, and the Longer History
To understand the Anthropic dispute in its full context requires stepping back further than January 2026, further even than the 2021 founding of the company. The surveillance infrastructure that Anthropic was seeking contractual protection against did not begin with AI. It began, in its modern form, decades earlier.
The NSA programs revealed by Edward Snowden in 2013 documented a surveillance architecture of staggering scope — PRISM, XKeyscore, bulk telephone metadata collection — operating largely in secret, authorized by a classified court whose proceedings were not subject to meaningful public oversight. Tech companies cooperated, voluntarily or under compulsion. Legal frameworks designed to permit targeted intelligence collection had been stretched to encompass something that looked, in practice, far more like the mass surveillance its architects denied conducting.
AI did not create this architecture. It is now positioned to exploit it in qualitatively new ways. The raw data collection capability already existed. What AI provides is the ability to make sense of that data — to classify, correlate, and act upon it — at speeds and scales previously impossible. The bottleneck between collection and exploitation is removed.
Research into autonomous weapons systems has been equally sobering. Academic studies published in 2024 and 2025 documented that machine learning is already enabling the substitution of autonomous systems for human soldiers across multiple battlefield roles, reducing what researchers call the "political cost" of offensive war. When autonomous weapons absorb casualties that would otherwise have been borne by human soldiers, the domestic blowback against military adventurism diminishes. Wars become politically easier to start.
The research further identified a profound governance failure: the rapid deployment of AI-augmented weapons systems has outpaced existing regulatory frameworks. Governance models designed for conventional autonomous weapons — typically rule-based and non-adaptive — were not designed to address the specific challenges of modern AI systems, including opacity (decision processes that cannot be interpreted by external observers), adaptivity (behavioral shifts in response to new inputs after deployment), and post-deployment drift (performance degradation over time in operational environments that differ from training conditions).
Research Finding
Generative and decision-support AI systems deployed in military contexts exhibit hallucinations, adversarial vulnerability, and misplaced confidence. These characteristics are particularly dangerous in high-stakes, real-time decision contexts where the consequences of confident wrong answers are irreversible. Unlike Bayesian or Dempster-Shafer systems, large language models have no principled mechanism for representing genuine ignorance — they generate fluent, confident output regardless of underlying epistemic uncertainty.
The United Nations Secretary-General, recognizing the trajectory, called for a legally binding treaty prohibiting lethal autonomous weapons systems from operating without human oversight — with a target completion date of 2026. The UN General Assembly voted in 2024 to begin formal negotiations. The International Committee of the Red Cross pressed for binding global accords. The Group of Governmental Experts under the Convention on Certain Conventional Weapons had been meeting since 2014 without producing enforceable constraints. Member states could not agree on basic definitions.
Meanwhile, the weapons were already in the field. The Russian Lancet loitering munition — a drone with AI-based autonomous targeting — had seen wide deployment in Ukraine. Estonian, South Korean, and American systems occupied a grey zone of partial autonomy that enabled states to operate within existing legal definitions while moving steadily toward full autonomous engagement authority. The governance frameworks were chasing deployments that had already happened.
Part VI The Human in the Loop
There is a counterargument to everything above, and it deserves honest engagement. Human beings in combat have always had to act on incomplete information, under conditions of extreme stress, with no time for deliberate analysis. The experienced sonar operator who says "that doesn't sound right" before he can formally articulate why is doing something genuinely sophisticated — integrating decades of pattern recognition in ways that formal mathematical frameworks cannot fully capture. Combat commanders have always exercised judgment that transcends any algorithm.
This is true. But it points toward a distinction that the autonomous weapons debate frequently obscures. The argument for human judgment in the loop is not an argument against AI assistance. It is an argument against the removal of moral weight and personal accountability from lethal decisions.
When a human being decides to fire — whether a torpedo officer authorizing a MK-48 ADCAP release, or a drone pilot authorizing a strike — that person owns the decision. They carry the moral weight of being wrong. They can be held accountable under military law, under rules of engagement, under international humanitarian law. That accountability may be imperfect. It may be evaded. But the framework exists, and the weight is real and personal.
When an algorithm fires, accountability dissolves. The engineer followed specifications. The commander deployed the system. The contractor met contract requirements. The program manager received certification. No single person made the decision. The decision was distributed across so many actors, so many procedural steps, so many institutional processes, that it effectively disappeared. This dissolution of accountability may not be an unintended side effect of autonomous weapons development. It may be precisely the point.
The USS Vincennes shootdown of Iran Air 655 in July 1988 — in which an experienced crew under stress misclassified a commercial airliner as a hostile military aircraft and killed 290 civilians — is often invoked as an argument for removing fallible humans from targeting decisions. But the lesson cuts both ways. The Vincennes incident produced accountability: investigations, congressional hearings, naval careers affected. It was documented, analyzed, and used to improve subsequent systems and procedures. The accountability, however imperfect, enabled learning.
An autonomous system that makes the same error produces no equivalent accountability. There is no one to investigate. The algorithm cannot testify. The training data cannot be cross-examined. The architecture cannot be held responsible under international humanitarian law. Since machines cannot be held responsible for breaches of the laws of war, any decision by an autonomous system must ultimately be traceable to a human — but the chain of traceability, in complex AI systems, may be genuinely impossible to reconstruct.
Epilogue - The Last Checkpoint
A checkpoint, in military engineering and in software systems, is a point at which execution pauses and human judgment is required before proceeding. Checkpoints are inefficient. They slow things down. They introduce the possibility of refusal. They are the architectural embodiment of accountability.
What the Pentagon sought from Anthropic — and what it ultimately obtained, in a different form, from OpenAI — was not merely unrestricted use of an AI tool. It was the removal of a contractual checkpoint: the company's ability to say, in writing, "this use is not permitted." Whether OpenAI's architectural safeguards will prove more durable than Anthropic's contractual ones remains to be seen. The Pentagon's under secretary of war characterized Anthropic's position as an attempt by a private company to "personally control the US Military." Anthropic characterized its position as a refusal to enable uses that current AI technology cannot perform safely.
Both characterizations contain truth. Both obscure it.
The deeper question — which neither the Pentagon nor Anthropic nor the courts have yet answered — is whether the checkpoint can survive at all. The surveillance infrastructure already exists. The autonomous weapons are already in the field. The governance frameworks are years behind the technology. The commercial pressures on AI companies to win government contracts are enormous, and the consequences of holding firm, as Anthropic discovered, are potentially existential.
Somewhere in the cold dark of a classified network, an AI system is processing data about human beings right now. It is generating confident outputs. It does not know what it does not know. It cannot tell the difference between a contact in its training distribution and one that is entirely novel. It will not hesitate.
The engineers who worried about CAPTOR mines were right to worry. They built systems that at least tried to represent their own uncertainty. They insisted on human beings in the loop at the moment of lethal decision. They understood that the known unknown was manageable and that the unknown unknown was the thing that should keep you awake at night.
What keeps us awake now is simpler and harder: the last checkpoint is disappearing, and nobody with the power to stop it seems certain they want to.
Sources & Formal Citations
Comments
Post a Comment