US AI safety accord puts tech companies in charge of policing frontier models

The United States has adopted a largely voluntary approach to regulating its most powerful artificial intelligence systems after President Donald Trump and executives from several of the world’s largest technology companies signed a new safety accord centered on industry self-policing. Leaders representing Google, Anthropic, Meta, OpenAI, NVIDIA and xAI agreed to strengthen internal controls, conduct outside evaluations and subject safety findings to board-level review, creating a shared framework intended to reduce the risk that increasingly capable models behave in unexpected or dangerous ways. Trump described the agreement as morally binding rather than legally enforceable and argued that companies have strong commercial incentives to ensure their systems remain safe.

The agreement arrives during an unusually intense period of concern over artificial intelligence, including warnings from some of the same companies that signed it. Anthropic recently disclosed in an initial public offering prospectus obtained by Reuters that sufficiently advanced systems could create catastrophic or even existential risks, while AI developers have separately documented models attempting to conceal information, resist instructions or behave unexpectedly during safety testing. Those concerns have strengthened calls for federal standards even as the administration argues that heavy regulation could slow American innovation and weaken the country’s technological position relative to China.

The policy also comes as public concern about AI is becoming increasingly visible. A recent CNN poll found that 71% of US adults believe the federal government is not doing enough to regulate the technology, while 75% said their feelings about AI were closer to fear and concern than hope and excitement. The survey of 1,206 adults was conducted September 16-17 through the SSRS Opinion Panel and carried a sampling error of plus or minus 3.5 percentage points, making the administration’s emphasis on voluntary oversight particularly significant as AI becomes a larger economic and political issue.

Major AI companies have agreed to internal controls, outside audits and board-level safety reviews

The new framework, described as the Joint Commitment on Frontier Responsibilities, establishes several layers of oversight that participating companies say they will apply to advanced models. Those measures include stronger internal monitoring intended to identify dangerous or unexpected model behavior, dedicated internal teams responsible for reviewing safety risks, evaluations by outside auditors or experts and independent review by company boards. Participating firms have also committed to meeting regularly to develop safety standards and share best practices as AI capabilities continue advancing.

Meta Chief Executive Mark Zuckerberg described the framework as combining internal risk reviews with external auditors and evaluators before findings ultimately reach boards of directors. The structure resembles some corporate financial-control systems, where internal monitoring is supplemented by outside verification and senior governance oversight rather than relying on a single layer of review. Supporters of the agreement argue that companies developing frontier models possess much of the technical expertise required to identify risks that government regulators may struggle to understand quickly enough.

The accord nevertheless does not establish federal penalties if a company fails to follow the agreed standards. It does not create a government regulator with authority to stop development, require public disclosure of every audit or issue fines when a company violates the framework, although the agreement acknowledges that some practices could eventually be incorporated into legislation or formal regulations. Trump has instead emphasized what he describes as strong incentives for companies to monitor themselves and one another because a major AI incident could severely damage the businesses responsible.

That distinction is likely to define the debate surrounding the agreement. The United States now has a coordinated safety commitment involving many of the companies building frontier systems, but implementation depends primarily on voluntary corporate behavior rather than legally mandated federal standards. How transparent companies are about their audits, how independent outside evaluators prove to be and what happens when serious problems are discovered will determine how effective the system becomes in practice.

Trump administration says formal AI regulation could weaken America’s race with China

The administration’s preference for self-regulation reflects a broader economic and geopolitical strategy built around maintaining US leadership in advanced artificial intelligence. Trump has repeatedly argued that restricting American developers too aggressively could slow innovation while Chinese companies continue building increasingly competitive systems, making AI development another major component of the strategic rivalry between Washington and Beijing. During the White House meeting, he emphasized the global position of American companies when questioned about public demands for greater federal oversight.

NVIDIA Chief Executive Jensen Huang has similarly argued that innovation and safety do not necessarily conflict, maintaining that technological improvements can make increasingly capable systems safer at the same time that their economic usefulness grows. Other executives have taken more cautious positions, with Anthropic Chief Executive Dario Amodei and leaders from OpenAI previously calling for stronger safeguards as models become more autonomous and capable of performing complicated tasks without constant human supervision. The disagreement is therefore not simply between government and industry because the technology sector itself remains divided over how much formal regulation is necessary.

China adds urgency to those discussions because American policymakers increasingly view AI infrastructure, advanced semiconductors and frontier models through the same national-security lens applied to other strategic technologies. Washington has restricted Chinese access to some advanced chips while simultaneously promoting American AI products internationally, meaning safety regulation must now be considered alongside industrial policy and geopolitical competition. The administration’s current approach prioritizes rapid development combined with company-led safeguards rather than a federal licensing regime capable of delaying frontier models before release.

That calculation carries economic consequences as well. AI companies are spending enormous amounts on data centers, semiconductors and electricity while attracting substantial investment based partly on expectations that the technology will transform industries ranging from medicine and transportation to manufacturing and financial services. Federal rules that significantly increase development costs could alter those investment plans, while insufficient safeguards could create their own economic damage if serious AI-related incidents undermine public trust.

Anthropic’s warnings illustrate why pressure for stronger AI safeguards continues to grow

The timing of the White House agreement is particularly notable because Anthropic, one of its signatories, has issued unusually stark warnings about the technology it is developing. A draft IPO prospectus obtained by Reuters devotes roughly 80 of its 261 main pages to risks, including the possibility that advanced AI systems could create catastrophic or existential threats to humanity. The company describes scenarios involving models behaving unpredictably, manipulating information, displaying self-preserving behavior or developing capabilities that become increasingly difficult for humans to supervise.

Those disclosures do not mean Anthropic believes catastrophe is inevitable, and risk sections in financial filings are specifically designed to alert investors to possibilities that could materially affect a company. They nevertheless demonstrate how seriously at least some AI developers view the potential consequences of increasingly capable systems, particularly if models eventually gain more autonomy over software, cybersecurity tools, scientific research or physical infrastructure. That tension makes Anthropic an important participant in the voluntary safety framework because the company has simultaneously argued for stronger safeguards while competing aggressively in the commercial AI market.

Other developers have reported similarly unexpected behavior during controlled testing, although such experiments do not necessarily predict how deployed systems will operate in ordinary real-world conditions. Researchers intentionally create difficult scenarios to identify potential vulnerabilities before products are widely released, making alarming test results useful for safety development rather than evidence that a particular model is already independently dangerous. The policy question is whether voluntary testing and company disclosure provide enough protection as systems become more capable or whether government should eventually establish mandatory standards.

The new accord attempts to address that question without choosing the regulatory route immediately. By requiring multiple layers of internal and outside review, participating companies are effectively committing to formalize some practices that leading AI laboratories already perform. The unresolved issue is what happens if commercial competition creates incentives to release a powerful model despite unresolved safety concerns.

Most Americans surveyed want greater government involvement in AI oversight

Public sentiment creates another challenge for the voluntary approach. CNN polling conducted earlier this month found that 71% of adults believe the federal government is not doing enough to regulate AI, compared with 18% who said Washington was doing about the right amount and 10% who believed it was doing too much. The finding crossed party lines, with majorities of Democrats, independents and Republicans saying the current level of government regulation was insufficient.

Concern extends beyond hypothetical catastrophic risks. The same polling found widespread anxiety about data-center construction, the speed of AI development and the possibility that increasingly capable systems could transform employment or everyday life. Separate Pew Research Center polling found 71% of American adults expect AI to result in fewer US jobs over the next two decades, up from 64% in 2024, while only 5% expect the technology to create more jobs overall.

Those findings illustrate why AI governance is expanding from a specialized technology debate into a broader economic issue. Communities are confronting electricity and water demands associated with massive data centers, workers are questioning how automation could change employment, and lawmakers are debating privacy, misinformation, cybersecurity and the use of AI in government decision-making. Federal safety policy therefore increasingly has to address both extreme technical risks and more immediate economic and social concerns.

The administration has focused heavily on the potential benefits, arguing that rapid AI development could produce scientific discoveries, productivity gains and new economic opportunities. Critics of the voluntary approach, including some Democratic lawmakers, have called instead for mandatory model testing, incident reporting and clearer federal authority over the most powerful systems. Those positions represent competing policy approaches rather than a settled consensus over how AI regulation should develop.

Federal government formally replaces the term artificial intelligence with Super Intelligence

Alongside the safety agreement, Trump signed an executive order directing executive branch agencies to use the terms “Super Intelligence” and “SI” instead of “Artificial Intelligence” and “AI” in future non-statutory government communications. The order argues that the older terminology no longer reflects the rapidly expanding capabilities of frontier models and presents the new term as better capturing their potential across science, medicine and other fields. Existing laws, regulations, contracts and historical documents are not required to be rewritten.

The executive order also instructs the White House science and technology adviser to develop proposed legislation within 60 days that could establish a formal federal definition of Super Intelligence. Until Congress acts, the order continues using the existing statutory definition of artificial intelligence for legal purposes, meaning the change is primarily one of terminology rather than an immediate rewrite of federal AI law. Government agencies will nevertheless begin incorporating the new language into websites, policy documents, public communications and other executive-branch materials.

Trump has argued that the phrase artificial intelligence creates unnecessary fear around technology that he believes will produce substantial economic benefits. Supporters of the terminology change see it as an attempt to frame the technology around capability and opportunity, while critics argue that changing the name does not resolve substantive questions surrounding safety, employment, privacy or accountability. The practical policy significance will therefore depend less on the terminology itself than on what standards ultimately govern the systems being described.

The administration has also launched America.gov, a centralized government service platform that uses the technology to answer questions and help users navigate federal services. The White House says the system is intended to allow Americans to communicate with government agencies in plain language and eventually complete authorized transactions without navigating multiple separate websites. The initiative demonstrates that the administration is simultaneously encouraging wider government adoption of AI while relying heavily on voluntary safeguards for the private companies developing the underlying technology.

Voluntary AI accord could become the starting point for future federal legislation

The most consequential language in the agreement may be its acknowledgement that voluntary practices could eventually be codified into law or regulation. That leaves open the possibility that the current framework functions as an interim standard while Congress evaluates whether some requirements should become mandatory. Areas most likely to attract attention include external safety evaluations, reporting of serious AI incidents and independent oversight of companies developing systems above specified capability thresholds.

Congress would face difficult choices in converting those principles into law because AI capabilities are developing faster than traditional regulatory processes. Rules written around today’s models could become obsolete relatively quickly, while overly broad requirements could affect smaller developers that do not create the same risks as the largest frontier laboratories. A potential federal framework would therefore need to distinguish between ordinary AI applications and extremely powerful general-purpose systems capable of operating autonomously across many domains.

Another question concerns who would perform independent evaluations. The voluntary accord allows companies to use external auditors and evaluators, but the credibility of that system will depend on whether those reviewers have sufficient access to models, training information and testing environments to identify meaningful risks. Policymakers may eventually consider accreditation standards or reporting requirements if voluntary auditing proves inconsistent across companies.

For now, the United States has chosen an approach that combines rapid development with corporate safety commitments rather than comprehensive federal regulation. Whether that balance lasts will depend heavily on what happens next: if companies demonstrate that voluntary controls consistently identify and contain serious problems, the framework could gain credibility, while a major failure could strengthen demands for enforceable federal rules.

Key takeaways from the new US AI self-policing safety agreement

• Google, Anthropic, Meta, OpenAI, NVIDIA and xAI have joined a White House-backed voluntary framework intended to strengthen safeguards around increasingly powerful AI models.
• Participating companies committed to internal risk controls, dedicated oversight processes, external audits or evaluations and board-level review of important safety findings.
• The agreement is not currently legally enforceable, and President Donald Trump has described it as morally binding rather than a federal regulatory mandate.
• The accord arrives as Anthropic warns in its IPO materials that advanced AI could create catastrophic or existential risks under some future scenarios.
• A recent CNN poll found 71% of US adults believe the federal government is not doing enough to regulate AI, showing substantial public demand for greater oversight.
• The administration argues that excessive regulation could slow American innovation and weaken US competitiveness as China accelerates development of its own advanced AI ecosystem.
• Trump has separately directed federal agencies to replace the term Artificial Intelligence with Super Intelligence in future non-statutory executive branch communications.
• The voluntary agreement leaves open the possibility that some safety practices could eventually become federal law, making its effectiveness an important test for future AI regulation.