Shapes

Filter and organize records

Remove duplicates

Removes repeated records, or tags them for review, using exact, fuzzy or phonetic matching.

Nodes › LogicOpen in Neblex
Good for
Clean, unique records
Role
Works on the records in a flow
Find it in
Nodes › Logic

When to use it

  • Sources contain the same customer or order more than once.
  • Near matches, such as spelling variants of a name, should be caught.
  • Duplicates should go to a review path instead of disappearing.

Reach for something else when

Merge by keyWhen duplicates describe the same thing and should be combined into one record.

Example

Before importing leads, remove duplicates by email and send possible fuzzy matches on company name to a review task.

  1. Read file
  2. Remove duplicates
  3. HubSpot

How it works

Settings
Compare records by field (several fields form a combined key; empty compares whole records). Match strategy: Exact match [default], Fuzzy - Jaro-Winkler, Fuzzy - edit distance (Levenshtein), Phonetic - Soundex. Similarity threshold (%) [90] for fuzzy strategies. Keep duplicates for branching.
Output
First occurrence wins. With Keep duplicates for branching every record is kept and tagged _duplicate (true/false) and _dup_of, and the node shows two outputs, Unique and Duplicate, to wire a review path.
Note
Exact matching is case-sensitive; fuzzy matching is not.
More in the guide

Add it to a flow

  1. Open the flow from Build › Workflows.
  2. Open Nodes and find Remove duplicates under Logic, or search for it. Click it or drag it onto the canvas.
  3. Drag from the previous step's output handle to its input, then open its settings with ✎ Edit.
  4. Run the flow and click the step to see the records it received and produced.

More in Filter and organize records

FilterKeeps only the records that match a condition and drops the rest.Nodes › LogicSortOrders records by a field, ascending or descending.Nodes › LogicKeep first NPasses on only the first records, up to the number you set.Nodes › LogicSplitTurns one record that holds a list into one record per list item.Nodes › LogicGroup into batchesPacks records into groups of a fixed size, so one call can carry many records.Nodes › LogicMerge by keyCombines records that share a key into a single record.Nodes › Logic