Get the latest tech news
Dbt – Incremental but Incomplete
Earlier this month, dbtTM launched microbatch incremental models in version 1.9, a highly requested feature since the experimental insert_by_period was introduced back in 2018. While it's certainly a step in the right direction, it has been a long time coming.
The initial microbatch implementation automatically filters models based on a user-specified column, lookback period, and temporal batch size (time granularity like day, month, year). The first solution is simply to track dates in state - something SQLMesh has supported from the jump - but this runs in direct contradiction to dbt's entrenched scripting / stateless design. SQLMesh can run an existing dbt project, adding various abilities like cron, state tracking, fine grained intervals (5 minutes - hour), batching/grouping, and parallelization.
Or read this on Hacker News