System Migration and Cutover

The problem

A migration is a one-time bulk move followed by weeks of both systems being live at once. The bulk move fails halfway and nobody knows which records made it. Then changes keep landing in the old system after the first pass, and the cutover happens on hope.

How it runs on Neblex

The first pass is a bulk flow: read the legacy records, map them to the new model with field mapping and lookups against reference Data Tables, remove duplicates on the natural key, validate every record, and write in checkpointed chunks. Rejected rows go to a branch that writes them to a Data Table with the reason, so the team fixes data rather than reruns the whole job.

While both systems are live, a Poll for changes flow keeps them aligned: a change detection key picks up inserts, updates, and deletes in the legacy system and applies them to the new one. For a record that changed in both places, sorting by modified time before a Merge by key step lets the newer version win, and a branch can hand a genuine conflict to a person as a task.

At cutover, a summary flow compares record counts and checksums on both sides and publishes the result. Every run is in the history with its inputs and outputs, so an auditor can trace any record from the old system to the new one.

Step by step

1. 1
   Map the modelField mapping and Data Table lookups translate legacy codes and picklists into the new system's values.
2. 2
   Bulk load in chunksThe initial pass runs in checkpointed chunks with duplicates removed and quality assertions before every write.
3. 3
   Quarantine the rejectsRows that fail validation are routed onward with the reason and written to a Data Table for cleanup.
4. 4
   Keep both sides alignedA change-detection flow applies ongoing inserts, updates, and deletes until the old system is switched off.
5. 5
   Cut over with evidenceA reconciliation flow compares counts and checksums and publishes the report before the switch.

Platform capabilities used

• Field mapping with Data Table lookups
• Chunked processing with checkpoints
• Remove duplicates
• Assert Data Quality
• Poll for changes with delete detection
• Merge by key
• Human review of conflicts with Wait for form
• Replay from the failed step

Common questions