科技圈的日常: “Perplexity 被发现使用隐形爬虫逃避 Robots.txt Cloudflare 发现 Perplexity 在使用隐形爬虫来绕过…”

Post #1392 14.3K Aug 4, 2025, 15:10 UTC

Perplexity 被发现使用隐形爬虫逃避 Robots.txt

Cloudflare 发现 Perplexity 在使用隐形爬虫来绕过 Robots.txt 与 WAF。

当 Perplexity 的爬虫无法正常爬取到网页内容后，会尝试使用未声明的 UA 和 IP 段来尝试绕过屏蔽规则。

Cloudflare 表示目前 Perplexity 的爬虫已经无法通过 Challenge 验证。

ref: https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/

Cloudflare Blog Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives Perplexity is repeatedly modifying their user agent and changing IPs and ASNs to hide their crawling activity, in direct conflict with explicit no-crawl preferences expressed by websites.

🥰 15
😁 4
❤ 3
👍 3
👎 2

Open in Telegram All posts from @misakatech