AI crawlers haven't learned to play nice with websites
SourceHut, an open source git-hosting service, says web crawlers for AI companies are slowing down services through their excessive demands for data.
"SourceHut continues to face disruptions due to aggressive LLM crawlers," the biz reported Monday on its status page. "We are continuously working to deploy mitigations. We have deployed a number of mitigations which are keeping the problem contained for now. However, some of our mitigations may impact end-users."
SourceHut said it had deployed Nep...
Read more at theregister.com