Get the latest tech news
Using watermarks to coordinate change data capture in Postgres
In change data capture, consistency is paramount. A single missing or duplicate message can cascade into time-consuming bugs and erode trust in your entire system. The moment you find a record missing in the destination, you have to wonder: is this the only one? How many others are there? Sequin
By examining each potential solution, we'll uncover the subtle pitfalls that led Sequin to adopt its coordination strategy. A few years ago, Netflix released a whitepaper on the design of their internal change data capture tool, DBLog. When it encounters a low watermark, it updates its state, indicating it should start saving PKs for the associated table OID.
Or read this on Hacker News