‘Extremely dystopian’: OpenAI CEO Sam Altman warns what could happen if powerful AI falls into wrong hands

OpenAI CEO Sam Altman has warned that the rapid development of artificial intelligence could go ‘very badly’ in two distinct ways: humans could lose control over powerful AI systems or too much power could become concentrated in the hands of a single person, company, country or AI lab.

Altman outlined the two risks in a lengthy post on X, arguing that AI development needs to follow a “narrow middle path” that prevents countries or companies from gaining too much control over the technology.

‘Team Humanity’ against loss of control

Altman said the first danger is that humanity could “lose control of the future to AI.” He called that outcome unacceptable and said OpenAI is “unapologetically on Team Humanity.”

According to Altman, AI must continue to serve people, and this requires safety and alignment techniques to advance quickly enough to keep pace with capable models.

His warning comes as AI systems become more capable and companies continue to compete to develop frontier models. Altman argued that safety cannot be treated simply as something that happens after a model is completed.

In a separate post, he said frontier AI companies have a responsibility to give the public confidence that capable systems will be developed responsibly.

The second danger: too much power in too few hands

On the other end of the spectrum, Altman pointed out another danger associated with AI: its highly concentrated power. According to Altman, there is a possibility of an extremely powerful AI controlled by one person or organization being used as a means of imposing that particular worldview on others, which would mean an extremely dystopian situation.

Moreover, this issue is not limited to companies but can be associated with countries as well. Thus, Altman believes that the task is to find the right balance between the advancement of AI and the prevention of its excessive concentration.

Altman is for slower, safer AI development

In his earlier post, Altman said companies should not wait for legislation or an antitrust exemption before working toward stronger AI safety standards. He said OpenAI is developing safety cases ahead of certain frontier reinforcement-learning runs that are expected to significantly increase model capabilities. He also called for shared standards covering areas such as misalignment, monitoring and safety.

Altman stressed that “pacing” AI development does not mean stopping progress. Instead, he explained in his tweet that progress should be slower than it otherwise could be when safety measures such as monitoring and safety cases require additional time and resources.

“No amount of American competitive pressure,” he argued, should justify recklessness or allow AI capabilities to move ahead of alignment and monitoring.

AI’s capability debate

Altman’s statement follows closely on the heels of Jacob Coxon, formerly an employee of Anthropic and OpenAI, resigning from Anthropic in reaction to the rapid development of artificial intelligence. Coxon warned in a massively viewed tweet that companies were moving towards developing self-improving superintelligences and stated that those making AI technology “earnestly believed” that it would kill mankind within the decade.

The CEO of Microsoft, Satya Nadella, claimed that the drive towards the development of artificial intelligence should come down to the extent to which the technology will benefit humanity and be controlled by mankind. He supports independent evaluators of artificial intelligence technology and, like Altman, believes that too much artificial intelligence power should not be held by a select few.