Fragmented Control, Collective Risk: Multi-Agent AI and the Coming Governance Crisis
Executive Framing: Context, Key Risks, and Outlook
Recent legislative frameworks such as the EU AI Act, the EU Data Act, and the Cyber Resilience Act (CRA) mark important milestones in the governance of artificial intelligence. However, these regulations (alongside evolving global AI policy efforts) fall critically short in addressing the growing risks posed by multi-agentic AI systems composed of multiple interacting AI agents capable of autonomous decision-making, self-organization, and goal pursuit. While existing laws largely focus on discrete AI applications, data governance, and product-level compliance, they are structurally unprepared for the emergent behaviors, coordination failures as well as compounded risks arising from agent collectives.
Multi-agentic AI introduces a new class of safety and alignment challenges. These systems range from swarms of autonomous drones to interacting digital agents in financial markets, cybersecurity operations, or online recommendation ecosystems. They can exhibit unanticipated dynamics such as collusion, runaway optimization, or emergent deception.
As multi-agent capabilities are scaled and networked (e.g., in edge computing, distributed robotics, or large language model collectives), they will produce complex socio-technical effects difficult to foresee or audit. Particularly alarming is the potential for competitive misalignment where agents, acting rationally under local incentives, generate globally unsafe or destabilizing outcomes. Such dynamics are not captured by current "risk tier" categorizations or conformity assessments in the EU AI Act.
Regulatory Gaps
The only country in the world to have a law specifically dedicated to agentic AI is China. In March 2025, the CAICT (China Academy of Information and Communications Technology), in collaboration with more than 20 Chinese AI companies (including Tencent, Alibaba and Huawei), released Part 1 of Technical Application Requirements for Intelligent Agents in Engineering (Frontier Technology Detector, 2025). Then, China’s ITU-T (International Telecommunication Union) published 748.46, Requirements and Evaluation Methods of Artificial Intelligence Agents Based on Large Scale Pre-Trained Models (ITU, 2025). This served to enhance performance of single-agent and multi-agent systems by assessing AI agent capability across a variety of tasks (perception, planning, execution and collaborative interaction).
Meanwhile, the West lacks national regulatory agentic governance. Instead, frontier model leading private enterprises and concerned academic-government sphere researchers are publishing recommended best practices (Patel, 2025). In December 2025, OpenAI co-founded the Agentic AI Foundation (AAIF) (OpenAI, 2025), alongside Anthropic and Block with the support of Google, Microsoft, AWS, Bloombery and Cloudflaire, to provide developer practices for governing agentic systems. AAIF was created because, “without common conventions and neutral governance, agent development risks diverge into incompatible silos that limit portability, safety and progress.”
The EU AI Act classifies risk based on individual system characteristics and use cases but lacks a framework for collective system behavior or inter-agent interactions. Similarly, the EU Data Act and CRA emphasize data sharing, cybersecurity, and resilience from a traditional software perspective, not accounting for autonomous decision-making in distributed AI collectives. There is minimal recognition of scenarios where multiple agents interact across organizational or jurisdictional boundaries, adapt in real-time, or influence each other via complex feedback loops. Even, the current Digital Omnibus package changes, which propose EU digital law harmonization and alignment, fail to address this oversight.
Technical Approaches and Proposed Contributions
Addressing these regulatory and safety gaps requires both policy innovation and advances in AI safety engineering. Technical directions include:
Multi-Agent Alignment Frameworks: New paradigms for ensuring cooperative behavior among autonomous agents, including mechanism design approaches, reward shaping for group alignment, and shared safety protocols.
Emergent Behavior Monitoring: Development of simulation environments and real-time telemetry tools to detect, predict, and intervene in collective agent behaviors before harmful outcomes emerge.
Teamwork and Human-AI Coordination: Techniques to enhance human oversight over distributed AI collectives, such as explainable multi-agent planning, shared mental models, and role-based control hierarchies.
Cross-Agent Accountability Protocols: Cryptographic and traceability tools to attribute responsibility and ensure auditability in agent networks, especially in dynamic or adversarial environments. These approaches should be incorporated into a new class of collective AI governance frameworks which are distinct from traditional product safety or data protection laws. Moreover, the field must explore domain-specific protocols. Especially in sectors where the scale and interdependence of AI agents pose sector-specific risks: finance, urban mobility, military defense, and synthetic media.
Implications and Open Questions
To ensure the safe deployment of increasingly autonomous and interactive AI systems, regulatory frameworks must evolve beyond a unitary AI system perspective. Multi-agentic AI represents a qualitatively different category of technological risk. This risk requires coordinated responses across technical, institutional, and international domains. Without substantial updates to both governance models and AI safety science, society risks entering a phase of unanticipated AI-driven systemic failures that no single actor, regulator, or agent can contain.
