- Genai can hallucinate the names of open source packages, experts warn
- Not always hallucinates a different name
- Cybercriminals can use the names to register malware
Security researchers have warned about a new method by which the generative AI (Genai) can be abused in cybercrime, known as ‘slopsquatting’.
It begins with the fact that the different Genai tools, such as Chat-GPT, Copilot and others, hallucinate. In the context of AI, “hallucination” is when AI simply invents things. It can form an appointment that a person never said, an event that never happened, or, in software development, an open source software package that was never created.
Now, according to Sarah Gooding of Socket, many software developers depend largely on Genai by writing code. The tool could write the lines itself, or could suggest different packages to the developer to download and include in the product.
Amazing malware
The report adds that AI does not always hallucinate a different name or a different package, some things are repeated.
“Upon executing the same activation request ten times, 43% of hallucinated packages were repeated every time, while 39% never reappeared at all,” he says.
“In general, 58% of hallucinated packages were repeated more than once in ten races, indicating that most hallucinations are not only random noise, but repeatable artifacts of how models respond to certain indications.”
This is purely theoretical at this time, but apparently, cybercriminals could draw the different packages that AI is hallucinating and – record them on open source platforms.
Therefore, when a developer receives a suggestion and visit Github, Pypi or similar, you will find the package and install it happily, without knowing that it is malicious.
Fortunately, there are no confirmed cases of slopsquatting in nature at the time of publication, but it is certain to say that it is only a matter of time. Since hallucinated names can be mapping, we can assume that security researchers will eventually discover them.
The best way to protect against these attacks is to be careful by accepting suggestions of any person, living or other.
