Get the latest tech news
Timeseries Indexing at Scale
Note This blog post was co-authored with May Lee and is cross-posted on the Datadog blog. Datadog collects billions of events from millions of hosts every minute and that number keeps growing and fast.
Kafka allows multiple services and teams to consume the same data stream independently and for different purposes, such as analysis, indexing, and archiving for long-term storage. We index timeseries points by their tags to make query execution more efficient and avoid scanning the data for the entire metric when only a small subset is requested. We learned that while in many cases it’s possible to make Go as fast as Rust, writing performance-sensitive code in Go requires a relatively larger time investment and deeper language expertise.
Or read this on Hacker News