Get the latest tech news

Notes on Binary Soup

I recently put together a small library called Slop, for intermediate on-disk data representation for the search engine, replacing a few ad-hoc formats I had in place before. This post isn’t so much an attempt to convince anyone else to use this library, as it makes trade-offs catering to a fairly niche use case, but to explore some of its design ideas, as it all came together very nicely, in the hopes that other libraries can draw ideas from it.

I recently put together a small library called Slop, for intermediate on-disk data representation for the search engine, replacing a few ad-hoc formats I had in place before. The storage format is designed to be trivially reverse engineered from the file-names themselves, in fact the hope is that anyone finding these files will be able to reverse-engineer an entire library for reading and producing them in short order, just from ls alone. The resulting code is a bit on the boiler-platy side, but as a trade-off the Lego sensibilities in the library also permits the implementation of ad-hoc projections and predicate pushdowns.

Get the Android app