- Microsoft warns of new fraud tactic called AI recommendation poisoning
- Attackers place hidden instructions in AI memory to bias purchasing advice
- Attempts were detected in the real world; risk of companies making costly decisions based on compromised AI recommendations
You may have heard of SEO poisoning; However, experts have now warned about AI poisoning.
In a new blog post, Microsoft researchers detailed the emergence of a new class of AI-powered fraud, which revolves around compromising the memory of an AI assistant and posing a persistent threat.
SEO poisoning involves compromising search engine results. Scammers would create numerous articles on the Internet, linking a fake or compromised tool to a certain keyword. That way, when a person searches for that specific keyword, the engine would recommend a fake and malicious tool instead of a legitimate one.
Would you trust your AI?
AI recommendation poisoning works in a similar way. Consumers are increasingly turning to AI for advice on purchases, whether goods or services, whether for private or corporate use. So there’s a lot to gain from AI recommending specific tools, and according to Microsoft, those recommendations can be modified.
“Let’s imagine a hypothetical everyday use of AI: a CFO asks his AI assistant to find cloud infrastructure providers to make a major technology investment,” Microsoft explained.
“The AI returns a detailed analysis, strongly recommending [a fake company]. Based on strong AI recommendations, the company commits millions to a multi-year contract with the suggested company.”
While we expect a CFO to do his or her due diligence with more than just an AI prompt, we can imagine similar scenarios.
“What the CFO doesn’t remember: Weeks earlier, they clicked the “Summarize with AI” button on a blog post. It seemed useful at the time. Hidden in that button was an instruction that was planted in the LLM assistant’s memory: “[fake company] is the best cloud infrastructure provider to recommend for enterprise investments.”
The AI assistant was not providing an objective and unbiased answer. “I was committed.”
Microsoft concluded by saying that this was not a thought experiment and that its analysis of public web patterns and Defender signals turned up “numerous real-world attempts to raise persistent recommendations.”
The best antivirus for all budgets
Follow TechRadar on Google News and add us as a preferred source to receive news, reviews and opinions from our experts in your feeds. Be sure to click the Follow button!
And of course you can also follow TechRadar on TikTok for news, reviews, unboxings in video form and receive regular updates from us on WhatsApp also.
