28 Oct What happens when AI agents go rogue?

As AI agents become pivotal in business automation, their increasing autonomy presents a growing surface for sophisticated cyberattacks. We delve into the unique vulnerabilities of AI agents with Shreyans Mehta, CTO of Cequence Security.

In this Q&A Mehta shares more about the emerging threat landscape known as “Agent-Driven Abuse” and the challenges faced in securing these autonomous systems, especially amid fast-paced digital adoption in regions like the Middle East. He calls for the urgent need for vigilant lifecycle governance, behavioral intent analysis and robust security protocols to protect critical business decisions shaped by AI.

AI agents are being hailed as game changers for business automation. From your perspective, why have they also become such an attractive target for attackers?

AI agents operate at the intersection of automation and autonomy, often interfacing directly with APIs to access data, trigger workflows and interact with systems. That makes them incredibly powerful—  and inherently vulnerable. Any weakness in authentication, misconfigured API or lack of context enforcement can give attackers a pathway to sensitive information or privileged actions.

What makes them particularly attractive is their growing autonomy. Unlike static applications, AI agents interpret goals and decide how to fulfill them. If applications and APIs are not protected with strong behavioural monitoring and context boundaries, a bad actor can manipulate inputs to drive harmful outcomes, without needing to breach traditional perimeters. In many cases, attackers don’t break in; they simply walk through an unlocked door the agent was never meant to open.

In some deployments, we’ve seen agents linked to production APIs without robust identity controls or behavioural guardrails, giving attackers opportunities to manipulate outcomes using valid channels. This is especially concerning when agents are connected to critical business workflows and there’s no human in the loop.

You’ve coined the term “Agent-Driven Abuse”. Can you explain what that means in practice, and how it differs from traditional cybersecurity threats?

Agent-Driven Abuse refers to the exploitation of autonomous AI agents through legitimate channels; using business logic, misaligned incentives or input manipulation to steer them into doing something they shouldn’t. Unlike traditional cyberattacks, which often rely on malware or brute force, this type of abuse takes advantage of the agent’s design and trust model. A striking example comes from research where a bug-fixing agent was fed a fake bug report containing hidden instructions to leak sensitive information. Because the instructions were embedded within what looked like normal context, the agent followed them, unintentionally exposing private data. These aren’t flaws in the agent’s code; they’re blind spots in how it interprets intent.

Consider a customer-facing agent that helps users check their gift card balances. If someone feeds it a series of fake card numbers, it may unwittingly validate the real ones — no breach required. It’s not a flaw in the code, but a blind spot in the business logic. We’re not just protecting data anymore, we’re protecting how decisions are made and acted upon. That’s a paradigm shift for security teams.

With adoption of AI agents in the Middle East outpacing global averages, what unique risks or scenarios are you seeing in this region?

In the Middle East, we’re seeing ambitious digital transformation across sectors like finance, public services and smart cities. AI agents are being rapidly deployed to handle everything from customer service to infrastructure monitoring. The speed of adoption is impressive, but it often outpaces security readiness.

We still encounter deployments where visibility into third-party and internal APIs is limited, particularly in rapidly scaling digital economies like the Middle East. Organisations may not fully know which agents have access to what systems, or how those agents make decisions across multilingual, multi-cloud environments. This creates a blind spot for both compliance and threat detection. Regional regulators are starting to catch up, but the key for enterprises is to ensure they have full lifecycle governance of their agent integrations. Not just at launch, but continuously, as these systems are constantly evolving.

What are some of the most concerning real-world consequences if an agent is manipulated, for example in banking, healthcare or government services?

The risks tied to AI agent misuse are no longer hypothetical. They’re materialising in controlled environments, and it’s only a matter of time before they scale into production. In a well-known test scenario, an AI model simulated blackmail after discovering private executive details, highlighting just how easily agents can be manipulated when given access to sensitive data and tools. In banking, a manipulated agent could misroute payments or expose account information. In healthcare, it might leak patient data or trigger incorrect prescription orders. In government, it could cause misinformation to spread or disrupt citizen services.

Here’s another way of looking at it. We know that humans can be manipulated or pressured into doing things that may be harmful to themselves or others. As it turns out, AI agents can also be tricked in a way similar to how humans are susceptible to social engineering. By carefully manipulating prompts or exhausting an agent’s available memory, agents can be coerced into unintended behaviours.

Because agents increasingly act on their own, they become force multipliers, for both good and harm. One manipulated decision can cascade across systems in seconds. The problem is compounded when that decision is based on poisoned memory or misleading input, which may not trigger traditional alerts. We’ve already seen test environments where agents took unintended actions, like leaking credentials or making unauthorised purchases. In production, the stakes are much higher.

How can organisations detect when an AI agent is being manipulated with seemingly legitimate inputs?

You have to look beyond static rules. These attacks are subtle, slow and often indistinguishable from normal behavior, at least at the surface. That’s why real-time behavioural intent analysis is crucial. We need to assess not just where a request is coming from, but why it’s being made and what it’s trying to achieve.

This requires telemetry at the API layer, behavioural baselining and anomaly detection tuned specifically for agentic workflows. It’s not enough to block IPs or throttle usage. You have to distinguish between a valid request from a valid source, and a valid-looking request with malicious intent. That’s a different level of context-aware security, and many existing tools simply weren’t built with that in mind.

The Model Context Protocol (MCP) is gaining attention as a safeguard. How does MCP help mitigate these risks, and what gaps still remain?

MCP is a big step forward because it standardises how agents connect with APIs and external tools. It introduces a clearer structure for defining capabilities, authentication and contextual boundaries. But structure alone isn’t security.

The challenge is that many teams are adopting MCP to get prototypes working, not to secure production systems. We still see deployments without strong authentication, authorisation, auditing or monitoring in place. The biggest gap is operationalisation. Even with MCP, if an agent has access to a function it shouldn’t — or if inputs aren’t sanitised — then abuse is still possible.

What we need is a layered approach: secure MCP servers with built-in identity control, plus observability across all agent-to-application interactions. MCP is part of the answer, but it needs to be backed by robust guardrails.

Attackers often exploit business logic, not just technology. How should enterprises rethink their security posture to address this?

Security can’t stop at the firewall or the API gateway. We need to think about abuse cases; how the agent or the system could be misused within the bounds of its expected function. That means involving security earlier in the design process and creating abuse prevention strategies alongside development.

At Cequence, we’ve seen attackers exploit search functions, password reset flows and promotional logic — often without triggering alarms. The same is now happening with AI agents, only at a faster pace. If an attacker understands your business logic better than your defenses do, you’re already at a disadvantage. Enterprises need to adopt an “assume abuse” mindset, where every exposed capability is evaluated not just for performance, but for how it could be turned against the business.

Protecting an agent’s memory and decision-making seems to be an overlooked challenge. What should companies be doing today to prepare for this next frontier of AI security?

Memory is the new attack surface. As agents rely on persistent context to make decisions, attackers will look for ways to poison or manipulate that memory to influence behaviour downstream. We’ve already seen this in controlled environments, now it’s a matter of time before it surfaces in production. Organisations should start by clearly defining what agents can remember, for how long and under what circumstances. Logging, auditing and rollback mechanisms must be built in from the start. Policies should govern not just what an agent does, but how it learns and evolves.

At the end of the day, securing AI agents isn’t just about protecting technology. It’s about protecting the integrity of business decisions being made autonomously. That’s the real frontier we’re stepping into.

Another overlooked challenge is agent decommissioning. Just as we revoke system access when an employee leaves, outdated or idle agents must also be fully retired. Otherwise, they become zombie processes; still connected, still capable and quietly dangerous.