Get the latest tech news

Kafka at the low end: how bad can it get?

There is oft-quoted advice that Kafka does poorly as a job queue. I’ve experienced this myself, and I wanted to formalize it a bit.

I’ll use the common architecture of a Web application submitting background jobs to workers via Kafka (for example, to generate a PDF of some report). Further, say you have 5 producers (Web application servers, here - Gunicorn processes, Kubernetes pods, whatever) that receive an API call and put a job onto this Kafka topic. This exact sequence of events is rare, but milder variations of this happen constantly when Kafka is used this way, at a low volume - such as only half, or three-quarters of your workers being busy, while the remainder are idle and there’s work queued, just sitting there.

Get the Android app