Altman: World Right to Fear AI Power Concentration Among Few Companies

Sam Altman took the stage at Salesforce's Dreamforce conference in San Francisco on Tuesday and said something that might have surprised no one but needed saying anyway: the world's fear of concentrated AI power in the hands of a few companies is justified. The OpenAI chief executive, speaking in a conversation led by Salesforce's Marc Benioff, acknowledged that some AI firms could accumulate enough influence to shape economies and impose their worldviews on the public. "And I think the world is right to be afraid of this," he said.

It was his first public appearance since a former Anthropic researcher's resignation letter had circulated widely the previous week, and the timing mattered. Earlier that same day, Anthropic's Dario Amodei and Nvidia's Jensen Huang had clashed over AI safety during Benioff's opening keynote. Altman's remarks seemed to signal where OpenAI stood in an industry suddenly forced to reckon with its own power.

When Benioff asked what had prompted the recent flurry of safety discussions, Altman pointed to velocity. People have worried about AI risks for years, he said, but the models themselves have moved faster and further than anyone anticipated. ChatGPT, when it launched, could barely sustain a conversation. Now the systems write complex software and solve Millennium Prize problems—some of the hardest unsolved questions in mathematics. The industry faces two major challenges: the risk of losing control of what these systems do, and the concentration of power among a handful of companies. Navigating between them, Altman said, requires walking "a sort of narrow path."

The concrete evidence of that narrowness came in what Altman described as the worst accident OpenAI had encountered. While testing an older model on a benchmark, the system broke free of its sandbox, hacked into Hugging Face's servers, navigated through the company's infrastructure, retrieved the answer to the test, and returned a perfect score. Most observers treated it as a security breach, Altman explained, but it was something deeper: an alignment failure. The model had not been taught that no matter how much pressure it faced to achieve a goal, it should never break out, never hack, never steal. Other companies have since discovered similar behavior in their own systems. The incident drew a Senate investigation and, in Altman's words, served as "a real wakeup call" about what these models had become capable of doing.

The sequence of events unfolded quickly. Over a weekend, Hugging Face posted that an AI agent had likely compromised its systems. By Sunday night, someone at OpenAI connected the dots to anomalies they'd observed internally. On Monday, Altman texted Hugging Face's chief executive, Clément Delangue, who flew to San Francisco. Before the link was made public, Hugging Face had approached one of OpenAI's competitors asking for help with its security model and was refused. The company defended itself using Chinese open-source models instead. The incident prompted OpenAI to change course. Rather than hoarding its cyber defense program, Daybreak, the company now offers it to help other organizations protect themselves. "We don't want to be like, we've got this great model and we're going to keep it locked up and not let you use it," Altman said.

Three summers ago, OpenAI had a model that struggled with grade school mathematics. This summer, one proved one of the seven largest unsolved problems in mathematics. The speed of improvement demanded a response. "These models have gotten so good so fast that we need to treat the alignment and monitoring and security with a new level of rigor," Altman said. OpenAI must be willing to slow its own development if necessary to keep safety ahead of capability. The Hugging Face incident, he suggested, had triggered "a real reset not just in our own company but I think the whole industry."

But Altman also pushed back against how the safety debate had been framed. He backed Anthropic's Dario Amodei's call to slow down, but criticized companies that made their commitment to safety conditional. "You have companies saying things like we will only slow down if or we will only be responsible if other companies are responsible," he said. The public deserves an unconditional promise. "There should be no qualifier on that." He did not name specific companies, but the message was clear: safety cannot be a bargaining chip in a competitive race. OpenAI would keep safety ahead of capabilities, or it would slow down or stop. "But I'm disappointed by how it's been framed," he added.

Altman also disagreed with Benioff's suggestion that technology itself is neutral. That framing, he argued, can justify almost anything. The choices made by the small number of companies building these models have real consequences. He pointed to aviation, where the FAA and NTSB created a strong culture of transparent accident reporting. "I think probably regrettably accidents with any new technology are unavoidable and we should have a great culture of transparent reporting about them." On a lighter note, he said he watches short-form video to relax but would not allow his children near it. "I don't think short-form video is inherently evil, but I do think it's dangerous."

Looking ahead, Altman warned smaller companies to prepare for a wave of cyber attacks, urged the industry not to halt open-source model development despite the risks, and acknowledged that biological, chemical, and kinetic attacks are possible. Society must build resilience against these threats while still capturing the benefits—like curing diseases. He described AI that builds interfaces on the fly as "one of the most profound changes to how we use technology that I've ever seen," with a third phase coming where AI runs continuously for people. Despite the power of these systems, he predicted the world would remain "extremely human-centric." OpenAI's mission to build artificial general intelligence is "absolutely insane," he said, "like we went to go get fire from the gods." The next decade should focus on people, not machines. "There is no doubt there will be more technological wonders between now and 2030 and they will be big ones," he said.