Get the latest tech news
Haydex: From Zero to 178.6B rows a second in 30 days
This is the story of how we turned a failed filter prototype into a production system running at 178.6B rows/sec in 30 days.
On May 16th, after days spent digging through a mountain of research papers on modern filter designs, I dropped a message into our team Slack channel: "The Grand Haydex Revival". Pool for our hash sets to aggressively recycle memory and added fast paths for ASCII string processing to avoid expensive UTF-8 machinery when it wasn't needed. The fast catalog lookups, the hyper-efficient hashing, the distributed indexing, the lazy block metadata loading, a new selective query ramp-up strategy, and a more robust filter normalization system.
Or read this on Hacker News