Bulk data processing that resumes from the failure, not from zero

Large migrations, nightly loads, and historical backfills usually mean reaching for a separate ETL tool, staging data somewhere in between, and hoping nothing fails halfway through.

When a batch job does fail at record 480,000 of 500,000, most tools make you start over from record one.

Bulk data processing in Neblex Integration Fabric moves large batches (nightly loads, migrations, backfills, bulk updates) natively, without staging them through a separate ETL tool.

Every step handles every record in the payload, and each step has a Repeat for each record setting that decides whether it works over the whole set at once or one record at a time. When a job needs explicit iteration, a Loop step runs its body for each record, a fixed number of times, or while a condition holds, with nested bodies and a maximum-iterations guard, and Run another flow calls a second flow as a nested run with its own history.

Turn on Repeat for each record when a later step has to act on each record, for example creating several rows for every row received. It then hands the next step one record at a time instead of the whole set. Either way, records move between steps in memory, so there is no staging bucket to provision and no data parked outside the run.

Chunked processing with checkpoints breaks a large batch into manageable pieces and tracks progress at the chunk level, so a failure partway through means resuming from the failed chunk, not restarting the whole job. A per-step concurrency cap processes several chunks at once without overwhelming a downstream system.

Data quality runs inline, not as a separate cleanup pass. A Remove duplicates step collapses rows on a key you choose, exact or fuzzy, an Assert Data Quality step checks row counts, required fields, numeric ranges, and any condition you write, and a record that fails can be routed onward with the reason attached to a branch you define, so one bad record does not stop the load.

When a destination slows down or starts returning rate-limit errors, the job honors the destination's own retry signals and backs off. Work is capped rather than buffered, and per-step record limits stop a runaway step before it exhausts memory. Transient errors are retried with backoff.

Because bulk jobs run on the same platform as everything else, the same transformation, notification, and audit steps are available as in any other flow.

To keep two systems continuously aligned rather than move one batch, that is data sync and change data capture. Bulk data processing is for the finite job with a start and an end.

Cleanup is part of the job, not a second tool you run afterward. Each step is optional and configured per job:

• Remove duplicates. Collapse rows on an exact key or a fuzzy match with a similarity threshold, first occurrence kept. Duplicates can be tagged and routed to a branch instead of dropped, and a Merge by key step produces one merged record when that is what you want.
• Assert Data Quality. Row count within bounds, required fields present, numeric ranges, and a condition evaluated on every record, all checked before anything is written.
• Normalize. Format date reshapes dates, Edit fields and Edit text reshape values, and a Lookup against a reference Data Table standardizes country and state codes.
• Route bad rows onward. A step's error handling passes a failed record with its reason down a branch you define, for example into a Data Table for review, while the load keeps going.
• Reconcile. Group and summarize produces source, loaded, and rejected counts at the end of the run.

Flows are built three ways, and you can move between them on the same flow:

• Natural language. Describe the flow to Matt, the Neblex AI Assistant built into the builder. Matt drafts the canvas, maps fields, and explains steps. It runs on the platform's builder model by default, or on a model you connect for the building experience only. That setting never supplies a model or credential to deployed agents, their tools, knowledge search, apps, or flows.
• Native builder. Assemble steps, branches, and connectors on the visual canvas.
• MCP. Create, change, publish, and run flows programmatically, so an external client or agent can build on Neblex directly, with consent scopes for operating, authoring, and publishing.

Agents, forms, apps, data tables, and connections have their own builders and the same MCP surface.

• Whole-set or per-record processing is a setting on each step, with a Loop step and Run another flow for explicit iteration
• Records move between steps in memory, so there is no staging bucket to manage
• Turn on Repeat for each record when a later step must act on each record, for example several rows created per row received
• Chunked processing with checkpoints: track progress at the chunk level, not just the job level
• Resume from the failed chunk on retry, instead of restarting the entire batch from zero
• A per-step concurrency cap, tuned to the downstream system's rate limits
• The destination's rate-limit signals are honored and work is capped rather than buffered, so a job never balloons in memory
• Inline data quality: Remove duplicates, Assert Data Quality, and error routing to a branch you define
• Group and summarize for source, loaded, and rejected counts at the end of every run
• Bulk jobs run as flows, sharing the same transformation, notification, and audit trail as any other automation
• Runs on the cloud runner or on on-prem workers, for jobs that must stay inside your network

A real job teams run every night, with the actual step-level detail.

Nightly load into Snowflake with dedupe, validation, and reconciliation

Every night, new and changed records from three source systems land in the warehouse. The job chunks the extract, cleans each chunk, loads what passes, quarantines what does not, and reconciles the counts before it reports done.

1. 1Extract: each source query returns the last 24 hours of new and updated records, by modified timestamp, from Salesforce, the billing database, and a product API, streamed in chunks.
2. 2Remove duplicates: rows are keyed on a natural key (email for contacts, order number for orders) and collapsed, with the duplicates tagged and routed to a review branch.
3. 3Assert Data Quality: required-field and numeric-range checks run per row; failures are routed onward with the reason into a Data Table.
4. 4Normalize: dates reformatted to UTC, and country and state values standardized through a lookup table.
5. 5Load: clean chunks are upserted into Snowflake under a per-step concurrency cap; when Snowflake throttles, the step honors its retry signal and backs off instead of buffering in memory.
6. 6Checkpoint: the checkpoint advances per chunk, so a failure at 2 a.m. resumes where it stopped, not at the first row.
7. 7Reconcile: a Group and summarize step compares extracted, loaded, and rejected counts and posts them to Slack with a link to the rejects table.