Cloudflare introduced a new
/crawl endpoint in its Browser Rendering service. The tool can scan and extract content from an entire website through a single API request.
The feature is designed for RAG pipelines, AI training datasets, monitoring, and research.
Cloudflare is also known as a major provider of bot protection for websites, including tools that block scrapers collecting data for AI models.
The company says its crawler will respect site rules and robots.txt.
📊 @tech