Get the latest tech news
Debugging distributed database mysteries with Rust, packet capture and Polars
Unravel a mysterious network bandwidth issue in QuestDB's primary-replica replication was identified and resolved. Learn about the tools and techniques used, including Rust for packet capture and Python with Polars for data analysis, to optimize network performance.
I hoped I could perform a simulated test run and capture a time series about packet sizes for both the inbound and outbound connections, but struggled to do this. After writing a script to generate some load, I set up my test with s3s-fs- a binary that emulates an AWS S3 endpoint, then ran a primary QuestDB instance on the same machine. Memory mapped files reduce the number of system calls and also allow the kernel to decide when to flush the data to disk in a very efficient manner.
Or read this on Hacker News