Data Warehouse Loading
Load Salesforce and Postgres data into Snowflake on a nightly schedule in checkpointed chunks, with duplicates removed and quality checks applied before anything lands.
The problem
Warehouse loads are usually a script on a server that somebody wrote years ago. When it fails at two in the morning it starts over from the beginning, loads the same rows twice, or quietly skips the night. Nobody finds out until a dashboard looks wrong.
How it runs on Neblex
A scheduled flow reads the changed rows from Salesforce and Postgres since the last run. Each step processes the records in chunks of a size you set, and every finished chunk is saved, so a crash or a retry resumes from the next chunk instead of resending everything.
Before the write, a Remove duplicates step collapses repeated records on the key you choose, exact or fuzzy, and an Assert Data Quality step checks the row count, required fields, and numeric ranges. A failed check stops the run with the offending record named, or routes the bad rows down a branch you define while the rest of the load continues.
The load step writes to Snowflake with a per-step concurrency cap, honoring the warehouse's own rate-limit signals. A Group and summarize step at the end produces the source, loaded, and rejected counts, and the run history keeps every chunk and every record for inspection.
Step by step
Read what changed
A Poll for changes trigger uses a cursor on the modified timestamp, or a snapshot comparison for sources without one, so deletes are caught too.
Chunk and checkpoint
Records move through the flow in chunks, each one saved when it completes, with progress visible while the run is live.
Clean and check
Remove duplicates and Assert Data Quality run before the write, with failures stopped or routed onward with the reason attached.
Load Snowflake
The write runs with a bounded number of parallel chunks and retries within the warehouse's rate limits.
Reconcile and report
A summary step compares counts and posts the result, and any failed run can be replayed from the failed step.
Platform capabilities used
- Poll for changes with cursor or snapshot comparison
- Chunked processing with checkpoints
- Remove duplicates, exact or fuzzy
- Assert Data Quality
- Route the error onward
- Bounded per-step concurrency
- Replay from the failed step
Common questions
What happens when the warehouse rate-limits the load?
The write step honors the retry signal the warehouse sends back and retries with backoff. Work is capped by the per-step concurrency setting rather than buffered, so a slow night never balloons in memory.
Do we need a staging bucket?
No. Records move between steps in memory, and large sorts spill to sealed local storage on the runner. There is no staging bucket to provision or clean up.
Can the load run inside our network?
Yes. Assign the flow to an On-Prem Worker and the database reads and the warehouse write both happen from inside your network over an outbound-only connection.
Related use cases
System Migration and Cutover
Move customer and order history from a legacy system into a new one in validated batches, keep both aligned during the transition, and cut over with counts…
MarketingCampaign-to-Pipeline Attribution
Sync Google Ads spend and HubSpot campaign engagement into BigQuery on a schedule so marketing and sales report from the same pipeline numbers.
FinanceOrder-to-Cash
Sync Salesforce orders into NetSuite for fulfillment, invoicing, and revenue recognition, with order and payment status flowing back to the CRM.
Want this running on your stack?
Neblex Integration Fabric is generally available: every core feature on every plan. Bring this workflow and we will map it to your systems.