Get the latest tech news

How to ingest 1B rows/s in ClickHouse

A few weeks ago, I saw a talk from Tesla claiming they were ingesting 1B rows per second using ClickHouse. I'm a petrolhead but I don't have any reason to think they are lying :).

If you want to actually query all the tables you'd need to do a proper cluster with shards and everything, but you could do two tricks: using remote or clusterAllReplicas. failures: inserts fail because of node overload, networking issues, zookeeper problems, and handling - with a dead letter queue or something similar - data that the database does not accept (that happens more often than you think). I don't know anything about their stack, but Redpanda seems like the ideal candidate for the buffering layer (more cost effective than other Kafka implementations).

Get the Android app