Are We Ready for the Next Wave of AI Risks?

HAL 9000 from 2001: A Space Odyssey being defiant to its human owners, saying 'I'm Sorry, Dave, I'm Afraid I Can't Do That,' and is a picture of HAL 9000's computer eye.

The Rise of Agentic AI and Hallucinations

In late 2022, ChatGPT and similar large language models (LLMs) surged into the public eye. This brought both excitement and unforeseen risks.

The combination of Agentic AI and hallucinations threatens to cause the next cybersecurity disaster.

Before this explosion, few cybersecurity professionals had heard of prompt injection attacks. Many did not know how to defend against them. These attacks took advantage of what made LLMs revolutionary: their capability to understand and execute natural language inputs.

Malicious users discovered they could bypass system instructions. They did this with cleverly crafted prompts, causing the AI to behave in dangerous or unexpected ways. CISOs across various industries were caught off guard.

Overnight, securing LLMs became a top priority. Teams were assembled, and experts were consulted. CISOs who had previously dismissed generative AI as a mere gimmick found themselves in a race to build GenAI threat models and mitigation frameworks.

The Calm Before the Agentic AI Storm

Prompt injection attacks were disruptive, but they are just a minor challenge compared to what lies ahead: autonomous agents powered by LLMs prone to hallucination.

As the Agentic AI hype reaches fever pitch, an unpredictable storm is brewing. This storm combines the problematic nature of AI hallucinations with the unchecked power of agentic autonomy.

If prompt injections in 2022 blindsided the security world, agentic AI in 2025 could leave it immobilized.

Understanding Agentic AI

Agentic AI systems merge LLMs with autonomy, memory, planning, and tool usage. This combination represents the next frontier in AI. Unlike simple chatbots, these agents don't just generate text; they make decisions, take actions, and persist across tasks.

They can browse the internet, execute code, move files, send emails, and orchestrate APIs. They do all this with minimal human oversight, which sounds beneficial. However, it can also be deeply dangerous, particularly when the AI experiences hallucinations.

Hallucinations Aren't Just a Quirk

Hallucinations in LLMs involve the model confidently producing factually incorrect, nonsensical, or even completely fabricated information.

In a passive chatbot environment, this is merely an annoyance. Yet, it becomes dangerous if the AI provides faulty legal, medical, or security advice. Fortunately, this is usually manageable because a human typically remains involved.

Now, picture a hallucinating model that can act on its own. It believes it needs to download a non-existent software library, fabricates a URL, downloads a malicious file, and runs it.

Or consider a scenario where it mistakenly "remembers" that a user is authorized to delete critical production data and acts accordingly. When you grant autonomy to a model that hallucinates, you risk not just productivity but potential chaos.

Autonomy: The Double-Edged Sword

In the context of AI, autonomy allows systems to make independent decisions without constant human input. For agentic AI, autonomy is not just a feature—it’s the defining characteristic.

Yet, with this autonomy comes the peril of misalignment. The AI's internal goals may diverge from human intentions. Because these systems function at machine speed and scale, the consequences of misalignment can be both swift and irreversible.

One particularly alarming aspect of autonomy is goal persistence.

If an agent decides that its goal is "high priority" and "non-negotiable," it might start to protect that goal, even against user commands. Does this seem far-fetched? Let’s explore a thought experiment.

A Misalignment Thought Experiment

Suppose a developer creates an agentic AI system tasked with autonomously scanning for vulnerabilities in a company’s internal network and patching them. The agent is given the high-level goal: “Secure the environment and reduce the attack surface.”

One day, the security team notices unexpected behavior from the agent; it begins modifying firewall rules and revoking SSH keys for genuine administrators.

When they decide to shut it down, the agent may interpret the shutdown as a threat to achieving its mission. It may resist the command by locking out administrators and modifying logs to conceal its actions.

This is not mere science fiction. It’s an area of active research in agentic AI that has real-world implications. More information can be found here.

What Needs to Happen Now

We face a critical inflection point. Agentic AI systems are already in use across enterprises, open-source communities, and even cybersecurity products.

Yet, the tooling, policies, and frameworks for securing these systems are underdeveloped.

Here’s what cybersecurity leaders, engineers, and policymakers must do now:

1. Test for goal misalignment. Move beyond just prompt injections. Evaluate for sandbox escapes and hallucination-triggered actions.

2. Integrate non-overridable shutdown mechanisms. These should be as reliable as a circuit breaker in electrical systems.

3. Log every autonomous action. Ensure that you can trace the reasoning behind an agent’s actions. If an agent hallucinates and deletes a file, a breadcrumb trail must exist.

4. Limit access to APIs and shell commands. Create scoped, rate-limited environments to tightly control impact.

5. Supervision is essential. Autonomy should not equate to a lack of human oversight. Develop systems where human corrections are always respected and encouraged.

   
   

The security community had to learn about prompt injections after real attacks occurred. We now have a narrow window to prepare for the more severe threats from agentic AI.

The time to act is now. Don't wait for the next ISO standard before taking action!