Skip to content
aiqbench takeAI Policy

aiqbench analysis

The page body is essentially empty, so this take rests on the title alone: it concerns how Perplexity's web crawling honors robots.txt, the standard that lets site owners control automated access. For AI readers, this bears on how AI search and answer engines source web content, and on the gap between a published crawler policy and what crawlers actually do. Verifying claims like this requires server logs and independent testing, not just a company statement.
Sources coveringPerplexity, The Verge
What this isA model's reading of the article and related coverage, written for aiqbench. Not the publisher's words.

How does Perplexity follow robots.txt?

Perplexity1 min read

How does Perplexity follow robots.txt?  Perplexity

Read the full article at Perplexity

Related coverage