High-Performance data transport in Rust on Linux: Putting madvise, mremap, and mmap to work, with optional iouring. Lightstream measured faster than Apache Arrow Flight (gold standard) on every axis in open 50gbps EC2 network benchmarks. Not supporting Windows was a pleasure.


Hi everybody,

Yesterday I released [Lightstream](https://github.com/spacecell/lightstream), a high-performance data transport library in Rust with Python bindings, that makes it effortless to send Apache Arrow, Protobuf, and Message Pack data over the network, shared memory, or piped out to the terminal.

During the development process I made extensive use of Linux sys call primitives including madvise, mremap, and mmap, and optionally enabled io\uring, for handling memory allocation efficiently, using zero-copy techniques. This included use of arena memory layouts to pack 'flatbuffers' next to each other, to help squeeze every ounce of performance out of the hardware. This differs from other libraries in the niche that tend to favour cross-system compatibility which I found was at the expense of performance, due to Linux's native capabilities.

It exceeded the performance of the gold standard industry comparison - Arrow Flight, on every axis of a 50gbps networking open benchmark, the details of which are attached and open to run in the Lightstream GitHub repository. This includes fully saturating each TCP connection thread, the NIC at 5.8GiB/s, and with p99 batch send time within 1% of p50 (I.e., stable).

For example, using 'mmap' , I was able to reach warm memory map reads from disk at 170GB/s, which is close to raw RAM speed.

If anyone here is big on this low-level hardware and software optimisation stuff, please feel free to ask any questions, I would be happy to discuss the architecture.

An excerpt of the comparison results are below.

Thanks,

Pete

Lightstream saturated the 50gbp\/s NIC with its decoding speed

Workload Shape

Mixed

|Streams|Arrow Flight GiB/s|Lightstream GiB/s|Ratio|
|:-|:-|:-|:-|
||
|1|0.939|1.109|1.18x|
|4|3.262|4.005|1.23x|
|8|5.138|5.677|1.10x|
|16|5.142|5.784|1.12x|

Numeric

|Streams|Arrow Flight GiB/s|Lightstream GiB/s|Ratio|
|:-|:-|:-|:-|
||
|1|0.649|1.109|1.71x|
|4|2.901|4.307|1.48x|
|8|4.851|5.693|1.17x|
|16|5.434|5.780|1.06x|

String Heavy

|Streams|Arrow Flight GiB/s|Lightstream GiB/s|Ratio|
|:-|:-|:-|:-|
||
|1|0.828|1.109|1.34x|
|4|3.052|3.861|1.27x|
|8|4.899|5.782|1.18x|
|16|5.217|5.790|1.11x|

Wide (100 cols)

|Streams|Arrow Flight GiB/s|Lightstream GiB/s|Ratio|
|:-|:-|:-|:-|
||
|1|0.695|1.108|1.59x|
|4|2.685|3.790|1.41x|
|8|4.549|5.725|1.26x|
|16|4.911|5.753|1.17x|

https://redd.it/1v63fh7
@r_linux