Get the latest tech news

Data Branching for Batch Job Systems

ly treated like code has been treated for decades. For many use-cases it isn't enough to know "What is the current value?" but also "What was the value previously?", "Who last changed the value?", and "Why did they change the value?".

For cases where we want to keep experimental data around for longer (i.e. not discard the output after a single job execution, but also not merge it to main) we can make use of experiment branches. An experiment branch can be longer lived that a single job execution, but isn't merged back into main. In this blog post I've shown how I've been thinking about data branches for different use-cases in a batch job-based software system.

Get the Android app