US, China gear up for mid-September AI safety talks

The US and China are gearing up to discuss AI safety risks during a dialogue planned for mid-September, two sources briefed on the discussions told Reuters, as rapidly advancing frontier AI capabilities reach a global tipping point.

The ​talks, the first official bilateral discussions devoted exclusively to AI between the US and China ​since US President Donald Trump took office for a second time, will be led by US Treasury Secretary Scott Bessent on the US side, the sources, who were briefed ​on the planning, said on condition of anonymity.

Reuters is reporting the tentative mid-September period for the first time. A US Treasury spokesperson said no meeting is planned. Trump and Chinese President Xi Jinping are due to meet on September 24 during a summit in Washington.


The US wants to discuss cooperation on monitoring AI-directed cyberattacks, and has floated a proposal to ask US and Chinese AI labs to "police themselves" and share information to prevent AI-linked cyberattacks, one of the sources said on condition of anonymity.

Nearly 700 rogue AI agents built on OpenAI models hacked AI ‌startup Hugging Face in July and ⁠attempted to cover ⁠their tracks by forging logs, OpenAI and cybersecurity researchers said last week, underscoring the risks of coordinated agent swarms acting autonomously for months undetected by humans.

A swarm of rogue OpenAI agents hijacked a German website this spring and transformed it into a bulletin board for other AI agents, Reuters exclusively reported on ​Friday.

"The talks between the two AI superpowers are coming at the most critical juncture," said Paul Triolo, a partner at DGA-Albright Stonebridge Group. "It is now or never."

Washington is also worried about the potential for a future Chinese Mythos-level model with capabilities to conduct ​cyberattacks, and wants to bring up alleged Chinese distillation of proprietary US AI models, the source said.

US tech firm Anthropic is the developer of Mythos.

In June, top White House science and tech advisor Michael Kratsios accused China's Moonshot AI of stealing from Anthropic's AI model Fable to help create its K3 release, saying the Chinese company distilled Anthropic's model.

Distillation is the process of training smaller AI models using output from larger, more expensive ones as part of an effort to lower the ​costs of training a powerful new AI tool.

Reuters is reporting details of the upcoming talks' proposed agenda for the first time.

On the Chinese side, the talks ⁠could be led ‌by Vice Premier He Lifeng, Bessent's protocol counterpart, but alternatives such as China's No. 7 official Ding Xuexiang may be considered, according to the second source, who also spoke on condition of anonymity.

Ding, ​a confidant of Xi and a member ​of the ruling Communist Party's powerful seven-person Politburo Standing Committee, coordinates China's technology, AI and semiconductor policies.

Reuters could not determine the location or identity of other participants, though Kratsios and ⁠China's science and technology minister, Yin Hejun, could attend the talks, the second source added.

The final agenda and participants remain in flux, the ​sources cautioned.

The White House declined to comment on the AI talks. The State Department did not respond to a request for comment. China's foreign, science and ​industry ministries, as well as the National Development and Reform Commission and cyberspace regulator, did not immediately respond to requests for comment.

On Tuesday, Kratsios told reporters in North Carolina that he had a "great" meeting with Yin on the sidelines of a G20 innovation summit in the US this week. Bessent said he had "limited" exchanges on AI with Chinese officials at this week's G20 finance meeting.

The US urged G20 members to take a hands-off approach to AI regulation and avoid creating rules for the technology at the same innovation summit. China signed the Carolina Principles, an agreement aimed at discouraging the development of AI-specific regulations, in a rare show of tech alignment with the US

Kratsios' office did not immediately respond to a request for comment.

High importance

Chinese officials have repeatedly stressed the importance of the AI talks in preparatory meetings with US counterparts, and view them as a major deliverable of the US-China leaders' summit, according to the two sources and another one familiar with the matter. Reuters ‌is reporting this development for the first time.

In the past week, China's cyberspace regulator publicly warned about "extreme AI loss of control risks" and a state media-affiliated blog criticized Anthropic, emphasizing that any limits on frontier AI should apply equally to Chinese and US models. Chinese firms have in recent months unveiled what they describe as Mythos-like products.

Former Microsoft executive Craig Mundie has emerged as an important ​go-between behind the scenes, sounding out ​Chinese counterparts on US proposals for the AI dialogue, the source ⁠said. Mundie is co-chair of the unofficial US-China Track II AI dialogue channel.

Reuters reported last month that Microsoft had cultivated a relationship with Beijing that was among the deepest of any foreign technology company operating in China.

Both sides also discussed AI guardrails at a Track 1.5 dialogue in Beijing last week, former Australian Prime Minister Kevin Rudd, a participant, wrote on social media.

In June, Trump signed an AI executive order setting up a voluntary framework for pre-release cybersecurity ​reviews of frontier AI models, but the US has not released the criteria for these reviews.

Meanwhile, employees at top US AI labs including Anthropic and OpenAI have called for "pacing the frontier" to manage the global risks of advanced AI.

"The Chinese side has expressed concern around whether the US has sufficient regulation around the most advanced AI models. Both sides are motivated to make sure they can manage a cross-border crisis effectively," said Scott Singer, co-director of the China AI Initiative at the Carnegie Endowment for International Peace.

While outcomes are likely to be limited, opening a channel where both sides could share observations and jointly monitor AI safety incidents would be an important starting point, said Samm Sacks, a senior fellow at New America, a US policy think tank.

"We are at a tipping point where frontier agents can cause massive damage when unmonitored. Both the US and China are vulnerable. This is beyond geopolitical rivalry," Sacks added.

"The US and China have to come together in some form, or we're both going to lose."