🚀 I Built a Production-Grade C++ Web Crawler That Hits 200+ Pages/Sec (Open Source!)

Hey Reddit! 👋

Spent the last month building a **high-performance web crawler in C++** that's actually production-ready. Think search engine infrastructure meets respectful crawling – fast, reliable, and doesn't piss off webmasters.

# 💪 Why this is different:

Most crawlers are either slow Python scripts or aggressive black boxes. This one's **built for scale** while being a good internet citizen. Perfect for building search indexes, ML datasets, content monitoring, or competitive intelligence.

# ⚡ Performance specs:

* **50-200+ pages/second** sustained (benchmarked on real workloads)
* **HTTP/2 multiplexing** with connection reuse and keep-alive
* **SIMD-accelerated parsing** using AVX2 for content processing
* **Lock-free work-stealing queues** across multi-threaded workers
* **RocksDB persistence** – survives crashes, graceful shutdown with state recovery

# 🤝 Respectful by design:

✅ Full **robots.txt compliance** – never crawls what it shouldn't
✅ **Per-domain rate limiting** with exponential backoff
✅ **Conditional GET** with ETag/Last-Modified caching
✅ Automatic **sitemap & RSS discovery**
✅ **Content deduplication** to avoid wasting bandwidth

# 🛠️ Built with modern C++:

* C++17 with proper concurrency primitives
* libcurl multi-interface (HTTP/2 ready)
* RocksDB for crash-safe metadata storage
* 16-partition priority queues + 256-shard rate limiter
* Tracy profiler integration for deep performance insights

# 📊 Real-world benchmarks:

|Setup|Workers|Throughput|
|:-|:-|:-|
|Small VPS (2 cores, 4GB)|6|\~30-50 pages/sec|
|Medium (8 cores, 16GB)|20|\~80-120 pages/sec|
|Beefy (16+ cores, 32GB+)|40+|150-200+ pages/sec|

# 🎯 Use cases:

* Building search engine indexes
* Training data for ML/NLP models
* Content change monitoring
* SEO competitive analysis
* Archive/research projects

# 🚀 Quick start:

bashcmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc)
./build/crawler 8 4 50000 --max-runtime 60

Fully documented with architecture diagrams, tuning guides, operational playbooks, and performance profiling tips. Would love feedback from anyone working on crawling infrastructure or dealing with large-scale web scraping! 🕷️

**GitHub:** [https://github.com/NotGyashu/Crawler](https://github.com/NotGyashu/Crawler)

https://redd.it/1o9ozzf
@r_cpp