Bulk data processing that resumes from the failure, not from zero

Large migrations, nightly loads, and historical backfills usually mean reaching for a separate ETL tool, staging data somewhere in between, and hoping nothing fails halfway through.

When a batch job does fail at record 480,000 of 500,000, most tools make you start over from record one.

Bulk data processing in Neblex Integration Fabric moves large batches (nightly loads, migrations, backfills, bulk updates) natively, without staging them through a separate ETL tool.

Every step handles every record in the payload, and each step has a Repeat for each record setting that runs it once per record instead, optionally in parallel. When a job needs explicit iteration, a Loop step runs its body for each record, a fixed number of times, or while a condition holds, with nested bodies and a maximum-iterations guard, and Run another flow calls a second flow as a nested run with its own history.

Turn on Repeat for each record when a step has to call something once per record, for example a per-customer API; up to 64 calls can run at once, and the next step receives the collected results. Records pass from step to step within the run, so there is no staging bucket to provision, and each step's results are kept in the run history under your retention settings.

Process in chunks breaks a large load on an adapter or HTTP step into pieces of 10 to 100,000 records and records each chunk as it lands, so a failure partway through means continuing from the first unfinished chunk, not restarting the whole job, and delivered records are not sent again.

Data quality runs inline, not as a separate cleanup pass. A Remove duplicates step collapses rows on a key you choose, exact or fuzzy, a Filter or conditional branch sends rows that miss a rule to a review path while the rest continue, and an Assert Data Quality step checks the batch for row counts, required fields, numeric ranges, and any condition you write before anything is written.

When a destination slows down or starts returning rate-limit errors, Retry on failure on the step tries again with backoff, and connectors that read the destination's retry signal wait as long as it asks. Record-by-record steps stream through in bounded pieces; steps that need the whole set, such as Sort, Remove duplicates, or Group and summarize, wait for every record first.

Because bulk jobs run on the same platform as everything else, the same transformation, notification, and audit steps are available as in any other flow.

To keep two systems continuously aligned rather than move one batch, that is data sync and change data capture. Bulk data processing is for the finite job with a start and an end.

Cleanup is part of the job, not a second tool you run afterward. Each step is optional and configured per job:

• Remove duplicates. Collapse rows on an exact key or a fuzzy match with a similarity threshold, first occurrence kept. Duplicates can be tagged and routed to a branch instead of dropped, and a Merge by key step produces one merged record when that is what you want.
• Assert Data Quality. Row count within bounds, required fields present, numeric ranges, and a condition evaluated on every record, all checked on the batch before anything is written. A failed check stops the run, or routes one error record to a branch.
• Normalize. Format date reshapes dates, Edit fields and Edit text reshape values, and a Lookup against a reference Data Table standardizes country and state codes.
• Route bad rows onward. A Filter or conditional branch sends rows that miss a rule down a review path, for example into a Data Table, while the rest of the load keeps going. With Repeat for each record and On error set to Route, a record the destination rejects is passed on with its reason instead of stopping the load.
• Reconcile. Group and summarize produces source, loaded, and rejected counts at the end of the run.

Flows are built three ways, and you can move between them on the same flow:

• Natural language. Describe the flow to Matt, the Neblex AI Assistant built into the builder. Matt drafts the canvas, maps fields, and explains steps. It runs on the platform's builder model by default, or on a model you connect for the building experience only. That setting never supplies a model or credential to deployed agents, their tools, knowledge search, apps, or flows.
• Native builder. Assemble steps, branches, and connectors on the visual canvas.
• MCP. Create, change, publish, and run flows programmatically, so an external client or agent can build on Neblex directly, with consent scopes for operating, authoring, and publishing.

Agents, forms, apps, data tables, and connections have their own builders and the same MCP surface.

• Every step handles the whole set; Repeat for each record runs a step once per record, up to 64 at once, with a Loop step and Run another flow for explicit iteration
• No staging bucket to manage: records pass from step to step within the run
• Process in chunks: progress recorded at the chunk level, not just the job level
• Continue from the first unfinished chunk on retry, instead of restarting the entire batch from zero
• Retry on failure with backoff for rate limits and transient errors
• Record-by-record steps stream in bounded pieces; whole-set steps wait for every record
• Inline data quality: Remove duplicates, Filter and branches for bad rows, Assert Data Quality on the batch
• Group and summarize for source, loaded, and rejected counts at the end of every run
• Bulk jobs run as flows, sharing the same transformation, notification, and audit trail as any other automation
• Runs in the cloud or on On-Prem Workers, for jobs that must stay inside your network

A real job teams run every night, with the actual step-level detail.

Nightly load into Snowflake with dedupe, validation, and reconciliation

Every night, new and changed records from three source systems land in the warehouse. The job chunks the extract, cleans each chunk, loads what passes, quarantines what does not, and reconciles the counts before it reports done.

1. 1
   Extract: each source query returns the last 24 hours of new and updated records, by modified timestamp, from Salesforce, the billing database, and a product API, streamed in chunks.
2. 2
   Remove duplicates: rows are keyed on a natural key (email for contacts, order number for orders) and collapsed, with the duplicates tagged and routed to a review branch.
3. 3
   Check rows: a Filter splits off rows that miss required fields or numeric ranges and writes them, with the reason, into a Data Table; an Assert Data Quality step then checks the remaining batch.
4. 4
   Normalize: dates reformatted to UTC, and country and state values standardized through a lookup table.
5. 5
   Load: clean rows are upserted into Snowflake with Process in chunks; when Snowflake throttles, Retry on failure waits and tries again.
6. 6
   Checkpoint: each chunk is recorded as it lands, so a failure at 2 a.m. continues where it stopped, not at the first row.
7. 7
   Reconcile: a Group and summarize step compares extracted, loaded, and rejected counts and posts them to Slack with a link to the rejects table.