Get the latest tech news
Alert Evaluations: Incremental Merges in ClickHouse
Oct 1, 2024 • 5 min read Alert Evaluations: Incremental Merges in ClickHouse Highlight.io is an open-source app monitoring platform, and our blog documents the infrastructure challenges we encounter while building our product. Interested in learning more? Check us out at https://highlight.io.
At Highlight, we rely on ClickHouse, an open-source columnar database built for handling large datasets and real-time analytics. ClickHouse's powerful -State and -Merge combinators, however, allow us to calculate intermediate values (in a similar way to the “Count” or “Sum” approach). Testing log alert evaluation in a real-world Highlight workspace, we were able to achieve a 10x speedup (1.24s to 0.11s) while bringing the memory usage down from 7.6 GB to 82 MB.
Or read this on Hacker News
