Data Warehouse Loading

The problem

Warehouse loads are usually a script on a server that somebody wrote years ago. When it fails at two in the morning it starts over from the beginning, loads the same rows twice, or quietly skips the night. Nobody finds out until a dashboard looks wrong.

How it runs on Neblex

A scheduled flow reads the changed rows from Salesforce and Postgres since the last run. Each step processes the records in chunks of a size you set, and every finished chunk is saved, so a crash or a retry resumes from the next chunk instead of resending everything.

Before the write, a Remove duplicates step collapses repeated records on the key you choose, exact or fuzzy, and an Assert Data Quality step checks the row count, required fields, and numeric ranges. A failed check stops the run with the violations named, and a Filter ahead of it can send rows that miss a rule down a branch you define while the rest of the load continues.

The load step writes to Snowflake in chunks, with Retry on failure for rate limits and transient errors. A Group and summarize step at the end produces the source, loaded, and rejected counts, and the run history records each chunk as it lands.

Step by step

1. 1
   Read what changedA Poll for changes trigger uses a cursor on the modified timestamp for new and changed rows, or a change detection key when deletes must be caught too.
2. 2
   Chunk and checkpointRecords move through the flow in chunks, each one saved when it completes, with progress visible while the run is live.
3. 3
   Clean and checkRemove duplicates and Assert Data Quality run before the write, with failures stopped or routed onward with the reason attached.
4. 4
   Load SnowflakeThe write runs with a bounded number of parallel chunks and retries within the warehouse's rate limits.
5. 5
   Reconcile and reportA summary step compares counts and posts the result, and any failed run can be replayed from the failed step.

Platform capabilities used

• Poll for changes with a cursor or a change detection key
• Chunked processing with checkpoints
• Remove duplicates, exact or fuzzy
• Assert Data Quality
• Route the error onward
• Bounded per-step concurrency
• Replay from the failed step

Common questions