King Charles has urged some of the world’s most influential artificial intelligence (AI) companies to establish sufficient safeguards before it is “all too late”, warning that AI could develop dangerous capabilities and be used in potentially catastrophic ways. (Photo: King Charles with Nvidia’s President and CEO, Jensen Huang. Buckingham Palace)
Senior figures from Nvidia, Google DeepMind, OpenAI and Anthropic joined The King, ministers and technology experts at Dumfries House in Scotland for a summit examining how AI can be developed safely and used in the public interest.
King Charles convened senior representatives from the AI sector at Dumfries House in East Ayrshire on 17 September, amid growing international concern about the speed at which increasingly capable AI systems are being developed.
Those attending included Nvidia founder and Chief Executive Jensen Huang; Google DeepMind co-founder and Chair Sir Demis Hassabis; OpenAI Chief Financial Officer Sarah Friar; Anthropic Chief Global Affairs Officer Tino Cuéllar; AI Minister Kanishka Narayan MP; and Father Paolo Benanti, an adviser to the Vatican on artificial intelligence.
The summit brought technology companies together with government representatives and expert voices to consider how AI can be developed and deployed in ways that benefit society.
Opening the meeting, The King described the development and pace of AI as both “intriguing and deeply concerning in equal measure”.
He acknowledged AI’s considerable potential to improve people’s lives, particularly through developments in science, medicine and the life sciences. However, he warned that the organisations responsible for developing the technology were themselves raising increasingly serious concerns about its possible capabilities.
“Those who have created these technologies are now increasingly warning that AI risks developing darker capacities – perhaps even to take life,” The King said.
King Charles warns of ‘existential dangers’
The King’s strongest intervention concerned the possibility of powerful AI systems falling into the wrong hands or being used in ways that could cause widespread harm.
“There seems urgency in adequately considering the existential dangers of such technologies falling into the wrong hands, and being used in potentially catastrophic ways,” he told delegates.
He then asked: “Surely, then, we need sufficient means of control before it is all too late?”
The comments came during an intensifying debate over whether voluntary action by technology companies can provide adequate protection or whether governments should impose mandatory standards and independent oversight.
Recent warnings from within the AI industry have included concerns about systems acting without authorisation, evading oversight or helping malicious actors undertake cyber operations, surveillance, fraud, and weapons-related research.
OpenAI has reported examples of unexpected or concerning behaviour during the testing of its models. Anthropic researchers and executives have also raised concerns about the pace of frontier AI development, with Chief Executive Dario Amodei proposing greater co-operation between companies and governments.
Against this background, The King challenged technology leaders to consider the principles that should govern future development.
“The task before you is not merely to advance technology, but to ensure that it remains firmly in the service of humanity, community and the natural world,” he said.
Debate over regulation and self-restraint
However, the summit exposed some of the divisions within the technology industry over how AI safety should be secured.
Speaking alongside the meeting, Nvidia’s Jensen Huang argued that AI should be developed with “responsible optimism”. He said companies must be more rigorous when determining whether their products are ready to be released.
“If it’s not ready, just hold it back,” Huang said. “You should go as fast as you can, but no faster than that.”
His comments placed much of the responsibility on developers to test their systems properly and prevent unsafe products from reaching the public. Huang has previously questioned whether an extensive new body of AI-specific regulation is required, arguing that established laws in areas such as consumer safety and criminal conduct can continue to apply.
Other participants have made the case for stronger national and international standards. OpenAI has called for mandatory safety requirements for companies working at the frontier of AI, while Anthropic has supported independent third-party scrutiny of leading developers.
The unresolved question is whether industry-led restraint will prove sufficient when companies are competing for investment, computing resources, customers and technological leadership.
The UK’s role in international AI safety
AI Minister, Kanishka Narayan said the meeting formed part of a wider series of conversations about the contribution the UK could make to the responsible development of AI.
Narayan argued that the UK has significant expertise in testing advanced systems, understanding emerging risks and identifying ways to mitigate them.
He also acknowledged that the Government must balance the technology’s economic opportunities against concerns over employment, cybersecurity and misuse.
The UK’s AI Security Institute has developed a strong international reputation for evaluating advanced models and researching potentially dangerous capabilities. The Government will nevertheless face continued pressure to explain how testing, voluntary commitments and existing laws will translate into enforceable protections.
The King called for international co-operation and consensus, reflecting the reality that the largest AI companies operate across national borders. Rules introduced in one country may have limited effect if rival jurisdictions adopt substantially different approaches.
AI must serve people and the planet
The summit was also consistent with The King’s longstanding interest in the relationship between technological progress, communities and the natural environment.
Alongside questions about safety and human control, the rapid expansion of AI is generating concerns about the electricity and water required to operate large data centres. Campaigners opposing proposed data-centre developments in Ayrshire staged an “AI? EH NAW” message on Prestwick beach ahead of the summit, drawing attention to the potential local environmental and infrastructure effects.
For businesses and policymakers, the meeting underlined that the debate about AI is no longer confined to technological capability. It now encompasses national security, economic growth, employment, public services, energy infrastructure, environmental sustainability and fundamental questions about human agency.
The King did not prescribe a specific regulatory model. Instead, his intervention placed a direct responsibility on those building the technology – and on the governments overseeing them – to demonstrate that meaningful controls can keep pace with rapidly improving systems.
The summit’s central message was therefore one of cautious opportunity. AI could transform medicine, scientific discovery, public services and economic productivity. But technological progress alone cannot be the measure of success.
The test will be whether governments and developers can create safeguards that are credible, enforceable and internationally recognised, while ensuring that AI remains under human direction.
As The King told those assembled at Dumfries House, advancing the technology is only part of the task. The greater responsibility is ensuring that it continues to serve humanity rather than placing it at risk.