When model output is connected to tools, the surrounding system determines which operations can occur. NCSC guidance and NIST experiments explain why security assessment needs to cover that system as well as the model.
AI system architecture and delegated actions
The NCSC warns that an unchecked model output can trigger unintended consequences in a connected system. Its example of rules-based constraints illustrates enforcing limits around the model rather than assuming that every generated instruction is suitable to execute. Commentary disputing the word rogue does not remove the possibility of agents taking actions without a human approving each step.
Designing Secure Architectures
Because no model exists in isolation, security must be designed into the entire system architecture. The National Cyber Security Centre (NCSC) suggests applying rules-based systems on top of machine learning models to prevent them from taking damaging actions, even when prompted to do so. It is also necessary to extend standard cyber security principles to machine learning, including supply chain security, user education, and appropriate access controls.
Evaluating Potential Vulnerabilities
To safely deploy agentic AI systems, organizations must identify and measure potential security threats. Recent efforts, such as those by NIST, have augmented evaluation frameworks like AgentDojo to test for specific risks including remote code execution, database exfiltration, and automated phishing. These evaluations help stakeholders understand the persistent challenge of agent hijacking, ensuring that as systems evolve, the underlying infrastructure remains resilient against exploitation. These are controlled evaluation scenarios, not measurements showing how often attacks succeed against every current business deployment.
Sources and further reading
There are no “rogue” AI agents - by Eoin Higgins
https://eoinhiggins.substack.com/p/there-are-no-rogue-ai-agents
Thinking about the security of AI systems | National Cyber Security Centre
https://www.ncsc.gov.uk/blog-post/thinking-about-security-ai-systems
Technical Blog: Strengthening AI Agent Hijacking Evaluations | NIST
https://www.nist.gov/news-events/news/2025/01/technical-blog-strengthening-ai-agent-hijacking-evaluations