News Score: Score the News, Sort the News, Rewrite the Headlines

Devs say AI crawlers dominate traffic, forcing blocks on entire countries

AI bots hungry for data are taking down sites by accident, but humans are fighting back. Software developer Xe Iaso reached a breaking point earlier this year when aggressive AI crawler traffic from Amazon overwhelmed their Git repository service, repeatedly causing instability and downtime. Despite configuring standard defensive measures—adjusting robots.txt, blocking known crawler user-agents, and filtering suspicious traffic—Iaso found that AI crawlers continued evading all attempts to stop t...

Read more at arstechnica.com

© News Score  score the news, sort the news, rewrite the headlines