Perplexity was recently accused of scraping sites that had explicitly disallowed LLM crawlers in their robots.txt files. In the wake of that revelation, a wave of how-to guides for blocking large-language-model scraping has surfaced [0]. They’re generally highly vitriolic, with people opposing this on both moral grounds (“AI is stealing your content”) as well as displaying a general distaste for AI.But how many of you wouldn’t hook up your website to Google?I know one of the primary reasons that...