Reddit has updated its robots.txt file, preventing Bing and many other search engines from crawling the site. “Bing stopped crawling Reddit after they implemented their updated robots.txt file on July 1, which prohibits all crawling of their site,” a Microsoft representative told Search Engine Land. What happened. On July 1, 2024, Reddit updated its robots.txt file preventing many search engines and AI tools from crawling the site. Reddit did not prevent Google from crawling the site, despite ...