Google warns that criminals are creating and selling illicit artificial intelligence tools, and the market is growing

AI Tools Are Being Created Specifically for Criminals, According to New GTIG Report
These tools bypass AI guardrails designed for security
‘Just in time’ AI malware shows how criminals are evolving their techniques

Google’s Threat Intelligence Group has identified a worrying shift in AI trends, as AI is no longer used just to make criminals more productive, but is now also developed especially for active operations.

Their research found that large language models (LLMs) are being used in malware in particular, with ‘just-in-time’ AI such as PROMPTFLUX, which is written in VBScript and ties into the Gemini API to request ‘specific VBScript obfuscation and evasion techniques to facilitate ‘just-in-time’ self-modification, likely to evade static signature-based detection.’

This illustrates how criminals are experimenting with LLMs to develop “dynamic obfuscation techniques” and target victims. PROMPTFLUX examples examined by Google suggest that this code family is currently in testing, so it could become even more dangerous once criminals develop it further.

Built to hurt

The market for legitimate AI tools is maturing, as is the criminal black market. Underground forums offer specially designed artificial intelligence tools that help lower the barrier for criminals to engage in illicit activities. This is bad news for everyone, as criminals no longer need to be particularly skilled to carry out complex cyberattacks and have an increasing number of options.

Threat actors are using tactics reminiscent of social engineering to bypass AI security features, posing as “cybersecurity researchers” to convince Gemini to provide them with information that might otherwise be prohibited.

But who is behind these incidents? Well, the investigation identifies, perhaps unsurprisingly, links to state-sponsored actors in Iran and China. These campaigns have a variety of objectives, from data exfiltration to reconnaissance, similar to state influence operations observed previously, which also use artificial intelligence tools.

Since AI tools became popular, both criminals and security teams have been using the tools to increase productivity and aid operations, and it’s not very clear who has the advantage.