Get the latest tech news

Astra Dynamic Chunks: How We Saved by Redesigning a Key Part of Astra

Introduction Slack handles a lot of log data. In fact, we consume over 6 million log messages per second. That equates to over 10 GB of data per second! And it’s all stored using Astra, our in-house, open-source log search engine. To make this data searchable, Astra groups it by time and splits the data…

While assuming fixed-sized chunks simplified the code, it also led to us allocating more space than required on our cache nodes, resulting in unnecessary spend. This helped ensure that we were able to pack the cache nodes as tightly as possible, resulting in a higher utilization of allocated space. It touched many key parts of Astra, essentially rewriting all of the logic that handled the assignment and downloading of chunks.

Get the Android app