BqLog is a cross-platform logging library open-sourced by Tencent two years ago. Since then it has gone through two more years of production use and improved a lot in performance, reliability, compatibility and developer experience. With 2.6.0 out, I'd like to introduce it here again.
BqLog is used in a number of Tencent's commercial products, including performance-sensitive mobile games such as Honor of Kings, as well as their server-side logic. It runs on more than 30 million devices every day, across a very wide range of hardware, networks and storage, so its stability and performance have held up in mature commercial products over a long time.
Design goals
Extreme speed is where BqLog starts, not where it stops. It is a mature, industrial-grade component, built for products that ship:
- Content to play a supporting role. On a modern machine a logging system should take as little of the system as it can. The logging threads and the consumer thread alike should use as little CPU, memory and disk I/O as possible. We do not optimize for benchmark scores; every optimization targets real use.
- Compression and encryption without giving up speed. The real-time compressed format is about 1/7 the size of text, and writing it is faster and uses less CPU than writing text. Hybrid RSA+AES encryption is supported and costs almost nothing in performance.
- Compatible everywhere it runs. Windows, macOS, Linux, iOS, Android, HarmonyOS, OpenHarmony and the Unix family; C++, Java/Kotlin, C#, Python, TypeScript/ArkTS and Go sharing one core and even one log object; game engines, mobile toolchains without an STL, strict
-Werror builds.- Built for the bad day. Logs written before a crash are recovered after it; a full disk neither crashes nor hangs the app; corrupted files are recovered as far as possible; log files can be encrypted to protect privacy.
- Care about the coding experience. One line to create a log, one line to write; the internals stay as transparent as possible, without complicated configuration; every interface is friendly to IDE code completion; integration tools such as UE Blueprints are provided.
Performance
2.6.0 comes with a new, complete benchmark against quill, fmtlog, spdlog, glog and Log4j2, and every result reports time, CPU and memory together. In short:
- Throughput: on macOS and Windows, BqLog has the shortest total time at every thread count. The compressed format is roughly 4 to 12 times faster than the fastest of the other libraries and also has the lowest CPU time; the text format is 1.5 to 4.6 times faster.
- Cost per call on the logging thread: BqLog, quill and fmtlog are in the same tier, with BqLog averaging under 20 ns per call on all three platforms; spdlog is more than an order of magnitude slower.
- Memory: with fixed-size buffers, BqLog's peak memory is in the lowest tier and barely grows with more threads; with growing buffers it is about half of quill's or less.
- File size: the compressed format is about 1/7 the size of text.
On Linux only latency is listed for now. On the VMs we tried, throughput was dominated by the VM's disk scheduling and varied by up to 2x between rounds, so we left it out. The test scripts are in the repo, and if you have a bare-metal Linux box we'd be glad to see your numbers.
Methodology, library configurations and full data: https://github.com/Tencent/BqLog/blob/main/docs/BENCHMARK.md
Technical write-ups
If you're curious why BqLog is fast, there are three articles:
1. High-Performance Realtime Compressed Log Format: https://github.com/Tencent/BqLog/blob/main/docs/Article%201Why%20is%20BqLog%20so%20fast%20-%20High%20Performance%20Realtime%20Compressed%20Log%20Format.MD
2. From Ring Buffer to Adaptive Data Bus: https://github.com/Tencent/BqLog/blob/main/docs/Article%202Why%20is%20BqLog%20so%20fast%20-%20From%20Ring%20Buffer%20to%20Adaptive%20Data%20Bus.MD
3.
