/b/ - Random

>Cloudflare, one of the biggest network internet infrastructure companies in the world, has announced AI Labyrinth, a new tool to fight web-crawling bots that scrape sites for AI training data without permission. The company says in a blog post that when it detects “inappropriate bot behavior,” the free, opt-in tool lures crawlers down a path of links to AI-generated decoy pages that “slow down, confuse, and waste the resources” of those acting in bad faith.

>Websites have long used the honor system approach of robots.txt, a text file that gives or denies permission to scrapers, but which AI companies, even well-known ones like Anthropic and Perplexity AI, have been accused of ignoring. Cloudflare writes that it sees over 50 billion web crawler requests per day, and although it has tools for spotting and blocking the malicious ones, this often prompts attackers to switch tactics in “a never-ending arms race.”

>Cloudflare says rather than block bots, AI Labyrinth fights back by making them process data that has nothing to do with a given website’s actual data. The company says it also functions as “a next-generation honeypot,” drawing in AI crawlers that keep following links to fake pages deeper, whereas a regular human being wouldn’t. It says this makes it easier to fingerprint malicious bots for Cloudflare’s list of bad actors as well as identify “new bot patterns and signatures” it wouldn’t have detected otherwise. According to the post, these links shouldn’t be visible to human visitors.

https://www.theverge.com/news/634345/cloudflare-ai-labyrinth-web-scraping-bots-training-data