Database Reference

Incremental loading

Load only what changed since your last run, instead of re-reading whole tables.

Filter on updated_at#

Use updated_at — not created_at — as your watermark. Orders change after creation (Processing → Shipped → Returned), and a created_at filter would never pick those changes up.

SQL

SELECT id, user_id, status, total_amount, created_at, updated_at
FROM orders
WHERE updated_at >= '2026-03-01T00:00:00Z'
  AND updated_at <  '2026-03-02T00:00:00Z'
ORDER BY updated_at, id;

Watermark pattern#

Store the highest updated_at you have loaded (your watermark) in your own state.
On each run, query rows with updated_at at or after the watermark. Using >= instead of > avoids missing rows that share the boundary timestamp.
Upsert on id into your destination so re-reading boundary rows is harmless.
Advance the watermark only after the batch is safely written.

Check indexing before relying on updated_at

created_at has an index, but updated_at may not. Without one, a filter on updated_at can require a full table scan, which is slow and may hit the 60 second statement timeout. Check with the data owner before relying on it for frequent incremental queries.

Initial backfill#

For the first load, walk forward through time in small windows rather than issuing one huge query, then switch to watermark-based runs.

Previous← Read-only role NextKnown limitations →