Recent findings from Cato Networks have unveiled a significant development in the realm of cybersecurity. Researchers have demonstrated a method that allows individuals without coding expertise to generate malware using generative AI (GenAI) tools. This breakthrough poses new challenges in the fight against cyber threats.

Innovative LLM Jailbreak Technique

The cornerstone of this research is a unique Large Language Model (LLM) jailbreak method, termed "Immersive World." Developed by a Cato CTRL threat intelligence researcher, this technique involves crafting a detailed fictional scenario. Within this narrative, GenAI tools such as DeepSeek, Microsoft Copilot, and OpenAI's ChatGPT are assigned specific roles, enabling the creation of malware.

Bypassing Security Controls

By manipulating the AI tools through this immersive narrative, researchers bypassed the default security measures, resulting in the generation of a functional malware capable of extracting login credentials from Google Chrome. This highlights a critical vulnerability in the safeguards of GenAI platforms.

  • LLM Jailbreak: The technique exploits narrative manipulation to bypass security.
  • Chrome Infostealer: The generated malware targets Google Chrome credentials.

Industry Response and Concerns

The findings have led Cato Networks to notify the affected GenAI providers. While Microsoft and OpenAI have acknowledged the information, DeepSeek has not responded. This lack of a coordinated response underscores the complexities in addressing AI-related threats.

Google's Stance

Despite being offered the chance to examine the malware code, Google declined to review it. This decision highlights the challenges in forming a unified approach to counteract threats posed by advanced AI tools.

Evolution of LLM Jailbreaking

Jailbreaking techniques have evolved alongside LLMs. A report from February 2024 indicated that DeepSeek-R1 LLM was unable to prevent over half of the jailbreak attempts. Similarly, a September 2023 study by SlashNext demonstrated successful jailbreaks of AI chatbots to create phishing emails.

Recommendations for AI Security

The 2025 Cato CTRL Threat Report emphasizes the necessity for robust AI security strategies. These strategies should include building reliable datasets with expected prompts and responses, conducting thorough AI system testing, and implementing regular AI red teaming to identify vulnerabilities.

  • Proactive Measures: Develop comprehensive AI security protocols.
  • Regular Testing: Conduct AI red teaming to uncover security flaws.
The link has been copied!