Get the latest tech news
Leader Election with S3 Conditional Writes
In distributed systems, for instance when scaling out some workload to multiple compute nodes, it is a common requirement to select a leader for performing a given task: only one of the nodes should process the records from a Kafka topic partition, write to a file system, call a remote API, etc. Otherwise, multiple workers may end up doing the same task twice, overwriting each other’s data, and worse.
That is, until last week, when AWS announced support for conditional writes on S3, which was received with great excitement by many folks in the data and distributed systems communities. When specified, the call will only succeed when no file with the same key exists in the target bucket yet; otherwise you’ll get a 412 Precondition Failed response. As Martin Kleppmann describes in this excellent post, checking lock validity and performing the leader’s actual work is not atomic, no matter how hard you try (for instance, think of unexpected GC pauses).
Or read this on Hacker News