News Score: Score the News, Sort the News, Rewrite the Headlines

Websites are Blocking the Wrong AI Scrapers (Because AI Companies Keep Making New Ones)

Hundreds of websites trying to block the AI company Anthropic from scraping their content are blocking the wrong bots, seemingly because they are copy/pasting outdated instructions to their robots.txt files, and because companies are constantly launching new AI crawler bots with different names that will only be blocked if website owners update their robots.txt. In particular, these sites are blocking two bots no longer used by the company, while unknowingly leaving Anthropic’s real (and new) sc...

Read more at 404media.co

© News Score  score the news, sort the news, rewrite the headlines