How we found a bug in the hyper HTTP library
None of the tools at the application level surfaced any errors, crashes, or log entries that provided useful clues. Application-level observability can have a blind spot for bugs that live below its awareness.
The failure occurred intermittently, scaled with response size, couldn’t be reproduced with simple tools like curl, and disappeared when we observed the system more closely. These signals pointed to a timing-dependent bug in the connection layer, not in the application logic.
Our breakthrough came from using kernel-level tooling with strace, the one layer that records what actually happened on the socket. The underlying bug lived in the few milliseconds between a partial flush and a premature shutdown — a window that opened only after we made the system faster.
Баг присутствовал в нескольких мажорных версиях hyper и, что иронично, проявился в Cloudflare из-за, в конечном счёте, оптимизаций архитектуры. А ещё это хорошее напоминание, что Rust защищает от гонок данных, но не от состояния гонок в целом.
