2025-07-23T15:02:00.000Many sites have been affected by the aggressiveness of web crawlers designed to improve LLMs.I’ve been relatively spared, but since the phenomenon started, I've been looking for a solution to implement.Today, I present a zip bomb gzip and brotli that is valid HTML.LLM Web CrawlersThe initial problem is the aggressiveness of LLM web crawlers that don't respect robots.txt. The first idea that comes to mind is IP blocking. However, web crawlers have circumvented this restrict...