- AI Robots and Scrapers Could Soon Overtake Human Users on the Internet
- Publishers Fight Declining Click Rates
- Robots.txt instructions are largely ignored
New data has claimed that AI bots are rapidly taking over web traffic, with data from Tollbit showing that there was one new visit from an AI bot for every 31 human visits in the final months of 2025, up from 1:200 in early 2025.
At the same time, and as humans interact more directly with AI, human visits fell by around 5% between the third and fourth quarter of 2025.
And it’s clear that human use of AI is responsible for this shift: while training crawls fell by about 15% between Q2 and Q4 2025, RAG bots rose by about 33%, and AI search indexers rose by about 59%.
The way we access the Internet is fundamentally changing
OpenAI leads the way in terms of scraping: its RAG bot ‘ChatGPT-User’ was around five times more active than Meta’s second most active bot, and around 16 times more active than Perplexity’s agent.
In terms of user behavior, a separate survey cited in the report found that more than a third (37%) of active AI users now initiate searches with artificial intelligence (such as ChatGPT or Gemini) instead of traditional search.
More alarmingly, the analysis found that robots.txt, a set of instructions to tell automated robots what parts of the page they can and cannot crawl, was ignored about 30% of the time on average, and up to 42% of the time by ChatGPT-User.
Since the robots.txt file is unenforceable and entirely dependent on goodwill, the instructions have been effectively deemed obsolete.
And that’s not the only bad news for publishers, who are struggling on the traffic front. Sites without direct AI licensing agreements saw click-through rates (CTR) drop around 3x between Q2 and Q4 2025. Even those with AI licensing agreements are not immune, and CTR rates also fell.
With all this at stake, Tollbit is calling on regulators to intervene to establish acceptable AI standards and protect intellectual property.
But with AI not going anywhere, it’s clear that the Internet is undergoing a major change and website owners should prepare for AI robots to become their primary readers.
Follow TechRadar on Google News and add us as a preferred source to receive news, reviews and opinions from our experts in your feeds. Be sure to click the Follow button!
And of course you can also follow TechRadar on TikTok for news, reviews, unboxings in video form and receive regular updates from us on WhatsApp also.
