AI-powered answer engine accused of evading bot detection and ignoring website preferences, prompting Cloudflare to take action
Cloudflare has raised serious concerns over the crawling behavior of Perplexity, an AI-powered answer engine, accusing it of using stealth tactics to bypass website no-crawl directives. According to Cloudflare’s latest findings, Perplexity has been observed modifying its user agent and source Autonomous System Numbers (ASNs) to obscure its identity and evade detection—despite websites explicitly disallowing such activity.
Initially, Perplexity crawled using a declared user agent. However, when blocked, it reportedly switched to undeclared agents and failed to fetch or respect robots.txt files, a standard protocol that communicates site owners’ preferences to bots. This behavior, Cloudflare asserts, violates long-standing norms of responsible bot operation.
“Crawlers must be transparent, serve a clear purpose, and follow the rules,” Cloudflare stated. “Perplexity’s tactics are incompatible with these principles, and as a result, we’ve de-listed them as a verified bot and implemented managed rules to block their stealth crawling.”
Cloudflare’s Verified Bots Policy outlines expectations for ethical bot behavior, including:
- Honest identification via unique user agents and declared IP ranges
- Respect for rate limits and robots.txt directives
- Separation of bots by activity type
- Clear public documentation of bot purpose
To protect its customers, Cloudflare has added signature matches for Perplexity’s stealth crawler to its managed rule set, available to all users—including those on free plans. Customers can choose to block or challenge suspicious traffic, ensuring only legitimate users gain access.
This development follows Cloudflare’s Content Independence Day initiative, which empowers over 2.5 million websites to block AI training crawlers. The company is also collaborating with global policy and technical experts, including the IETF, to standardize extensions to robots.txt and reinforce ethical bot behavior.
As AI-driven services proliferate, Cloudflare’s move signals a growing demand for accountability in how bots interact with web content. The message is clear: transparency and respect for digital boundaries are non-negotiable in the age of intelligent automation.
