Get the latest tech news
Multi-tenant queues in Postgres
An unfair advantage: multi-tenant queues in Postgres Alexander Belanger Published on April 18, 2024 TL;DR - we've been implementing fair queueing strategies for Postgres-backed task queues, so processing Bob's 10,000 files doesn't crowd out Alice's 1-page PDF. We've solved this in Hatchet (opens in a new tab) and Hatchet Cloud (opens in a new tab) so you don't have to — here's a look at how we did it.
You can find all the code used in these examples — along with some nice helper seed and worker commands — in this repo: github.com/abelanger5/postgres-fair-queue (opens in a new tab). If you'd prefer a higher write throughput, setting a small limit for placing tasks in the OVERFLOW queue and using the partition method from above may be a better approach. But there are other approaches which provide approximate fairness, such as shuffle sharding (opens in a new tab), which we'll show how to implement in Postgres in a future post.
Or read this on Hacker News
