At the Seoul AI Summit, co-hosted by the Republic of Korea and the United Kingdom, international organizations met to discuss the global advancement of artificial intelligence.
Participants included representatives of the governments of 20 countries, the European Commission and the United Nations, as well as leading academic institutes and civil groups. Several AI giants were also in attendance, including OpenAI, Amazon, Microsoft, Meta, and Google DeepMind.
The conference, which took place on May 21 and 22, followed the AI Safety Summit, held in Bletchley Park, Buckinghamshire, UK, last November.
One of the key objectives was to move towards the formation of a global set of AI safety standards and regulations. To achieve this, a series of key measures were adopted:
- The tech giants have committed to publishing security frameworks for their cutting-edge AI models.
- The nations agreed to form an international network of AI safety institutes.
- The nations agreed to collaborate on risk thresholds for cutting-edge artificial intelligence models that could help in building biological and chemical weapons.
- The UK government is offering up to £8.5 million in research grants to protect society from the risks of AI.
UK Technology Secretary Michelle Donelan said in a final statement: “The agreements we have reached in Seoul mark the start of Phase Two of our AI Security agenda, as the world takes concrete steps to become more resilient to AI risks and begins to deepen our understanding of the science that will underpin a shared approach to AI safety in the future.”
1. Tech giants committed to publishing security frameworks for their cutting-edge AI models.
16 global AI companies have agreed to new voluntary commitments to implement best practices related to AI security at the border. Frontier AI is defined as highly capable general-purpose AI models or systems that can perform a wide variety of tasks and match or exceed the capabilities present in the most advanced models.
The undersigned companies are:
- Amazon (United States).
- Anthropo (USA).
- Coheré (Canada).
- Google USA).
- G42 (United Arab Emirates).
- IBM (United States).
- Inflection AI (USA).
- Goal (United States).
- Microsoft (United States).
- Mistral AI (France).
- Naver (South Korea).
- OpenAI (USA).
- Samsung Electronics (South Korea).
- Institute of Technological Innovation (United Arab Emirates).
- xAI (United States).
- Zhipu.ai (China).
The so-called Frontier AI Security Commitments promise the following:
- Organizations effectively identify, assess and manage risks when developing and deploying their cutting-edge AI models and systems.
- Organizations are responsible for safely developing and deploying their cutting-edge AI models and systems.
- Organizations' approaches to AI border security are adequately transparent to external actors, including governments.
The commitments also require these technology companies to publish security frameworks for how they will measure the risk of the frontier models they develop. These frameworks will examine the potential for misuse of AI, taking into account its capabilities, safeguards, and implementation contexts. Companies should note when serious risks would be “considered intolerable” and highlight what they will do to ensure thresholds are not exceeded.
SEE: Definition of generative AI: how it works, benefits and dangers
If mitigations do not keep risks within thresholds, the undersigned companies have agreed “not to develop or implement (the) model or system at all.” Their thresholds will be announced before the AI Action Summit in France, scheduled for February 2025.
However, critics argue that these voluntary regulations may not be strict enough to substantially impact the business decisions of these AI giants.
“The real test will be how well these companies deliver on their commitments and how transparent they are in their security practices,” said Joseph Thacker, principal artificial intelligence engineer at security company AppOmni. “I didn't see any mention of consequences and aligning incentives is extremely important.”
Fran Bennett, acting director of the Ada Lovelace Institute, told The Guardian: “For companies to determine what is safe and what is dangerous, and voluntarily choose what to do about it, that is problematic.
“It's great to think about security and setting standards, but now you need some muscle: you need regulation and some institutions that are able to draw the line from the perspective of the people affected, not the companies that build the things.”
2. Nations agreed to form an international network of AI safety institutes
World leaders from 10 countries and the EU agreed to collaborate on AI safety research by forming a network of AI Safety Institutes. Each of them signed the Seoul Declaration of Intent for International Cooperation in AI Safety Sciences, which states that they will foster “international cooperation and dialogue on artificial intelligence (AI) in the face of its unprecedented advances and the impact on our economies and societies.”
The nations that signed the declaration are:
- Australia.
- Canada.
- European Union.
- France.
- Germany.
- Italy.
- Japan.
- Republic of Korea.
- Republic of Singapore.
- United Kingdom.
- United States of America.
The institutions that will form the network will be similar to the UK's AI Safety Institute, which launched at the AI Safety Summit in November. It has three main objectives: evaluate existing AI systems, conduct fundamental research on AI safety, and share information with other national and international actors.
SEE: UK AI Safety Institute launches open source testing platform
The United States has its own AI Safety Institute, which was formally established by NIST in February 2024. It was created to work on priority actions outlined in the AI Executive Order issued in October 2023; These actions include the development of standards for the security of AI systems. South Korea, France and Singapore have also created similar research centers in recent months.
Donelan credited the “Bletchley effect” (the formation of the UK AI Safety Institute at the AI Safety Summit) for forming the international network.
In April 2024, the UK government formally agreed to work with the US on developing tests for advanced AI models, largely sharing developments made by their respective AI Safety Institutes. The new Seoul agreement provides for the creation of similar institutes in other nations that join the collaboration.
To promote the safe development of AI globally, the research network:
- Ensure interoperability between technical work and AI security by using a risk-based approach in the design, development, implementation and use of AI.
- Share information about the models, including their limitations, capabilities, risks, and any security incidents they are involved in.
- Share best practices on AI security.
- Promote sociocultural, linguistic and gender diversity and environmental sustainability in the development of AI.
- Collaborate on AI governance.
AI Safety Institutes will have to demonstrate their progress in testing and evaluating AI safety at next year's AI Impact Summit in France, so they can advance discussions on regulation.
3. The EU and 27 nations agreed to collaborate on risk thresholds for frontier AI models that could help in the construction of biological and chemical weapons.
Several nations have agreed to collaborate on developing risk thresholds for border AI systems that could pose serious threats if misused. They will also agree when model capabilities could pose “serious risks” without appropriate mitigations.
These high-risk systems include those that could help bad actors access biological or chemical weapons and those that have the ability to evade human oversight without human permission. An AI could achieve the latter through safeguard circumvention, manipulation, or autonomous replication.
The signatories will develop their risk threshold proposals with AI companies, civil society and academia and discuss them at the AI Action Summit in Paris.
SEE: NIST establishes AI security consortium
The Seoul Ministerial Declaration, signed by 27 nations and the EU, binds countries to similar commitments made by 16 AI companies that accepted Frontier AI Safety Commitments. China, in particular, did not sign the declaration despite being involved in the summit.
The nations that signed the Seoul Ministerial Declaration are Australia, Canada, Chile, France, Germany, India, Indonesia, Israel, Italy, Japan, Kenya, Mexico, Netherlands, Nigeria, New Zealand, Philippines, Republic of Korea, Rwanda. Kingdom of Saudi Arabia, Singapore, Spain, Switzerland, Türkiye, Ukraine, United Arab Emirates, United Kingdom, United States of America and European Union.
4. The UK government is offering up to £8.5 million in research grants to protect society from the risks of AI.
Donelan announced that the government will provide up to £8.5 million in research grants to study the mitigation of AI risks such as deepfakes and cyberattacks. Recipients will work in the area of so-called “systemic AI security,” which seeks to understand and intervene at the societal level at which AI systems operate, rather than the systems themselves.
SEE: 5 deepfake scams that threaten companies
Examples of proposals eligible for a Rapid Grant for Systemic AI Safety could be considered:
- Stop the proliferation of false images and misinformation by intervening in the digital platforms that spread them.
- Prevent AI-based cyberattacks on critical infrastructure, such as those that provide energy or healthcare.
- Monitor or mitigate potentially harmful side effects of artificial intelligence systems that take autonomous actions on digital platforms, such as social media bots.
Eligible projects could also cover ways that could help society reap the benefits of AI systems and adapt to the transformations it has brought about, for example through increased productivity. Applicants must be based in the UK, but will be encouraged to collaborate with other researchers around the world, potentially associated with international AI safety institutes.
The Fast Grant programme, which expects to offer around 20 grants, is led by the UK AI Safety Institute, in partnership with the UK Institute for Research and Innovation and the Alan Turing Institute. They are specifically looking for initiatives that “offer concrete and actionable approaches to the significant systemic risks of AI.” The most promising proposals will become longer-term projects and may receive additional funding.
UK Prime Minister Rishi Sunak also announced the 10 finalists for the Manchester Prize, with each team receiving £100,000 to develop their AI innovations in energy, environment or infrastructure.
