feat(core): Dedupe #10101

ShireenMissi · 2024-07-18T16:40:26Z

Summary

Building on the work done in #5178
This PR creates a new version of Remove Duplicate node with the ability to remove duplicates across executions. it supports three logic types:

"Remove Items With Already Seen Key Values": Dedupe based on one or more keys, the whole value is hashed and stored.
"Remove Items Up to Stored Incremental Key": Dedupe based on incremental key, only store the. highest value processed previously.
"Remove Items Up to Stored Date": Dedupe based on a date key, only store the latest date processed.

the deduplication happens in two contexts Node and Workflow.

Related Linear tickets, Github issues, and Community forum posts

https://linear.app/n8n/issue/NODE-1487/dedupe-p0

Review / Merge checklist

PR title and summary are descriptive. (conventions)
Docs updated or follow-up ticket created.
Tests included.
PR Labeled with release/backport (if the PR is an urgent fix that needs to be backported)

tomi

Thank you for addressing all the comments 💟 Couple comments mainly about code readability that would be nice to address and a question about the sorting. But otherwise LGTM 🚀

tomi · 2024-10-09T16:35:27Z

packages/cli/src/deduplication/deduplication-helper.ts

+		items: DeduplicationItemTypes[],
+		mode: DeduplicationMode,
+	): DeduplicationItemTypes[] {
+		return items.slice().sort((a, b) => (DeduplicationHelper.compareValues(mode, a, b) ? 1 : -1));


The sorting is not currently stable (i.e. equal items don't have guaranteed order). Does it need to be? If it does, we should return 0 when the items are equal

it took me a while to get :) and it was actually an issue, so I fixed it

tomi · 2024-10-09T16:37:02Z

packages/cli/src/deduplication/deduplication-helper.ts

+		return createHash('md5').update(value.toString()).digest('base64');
+	}
+
+	private async fetchProcessedData(


Personally fetch sounds like it's making an http request (because of fetch api). Maybe we could rename this queryProcessedData or selectProcessedData?

I changed it to findProcessedData to convey that it is a wrapper for findOne

packages/nodes-base/nodes/Transform/RemoveDuplicates/utils.ts

tomi · 2024-10-09T16:46:03Z

packages/workflow/src/Interfaces.ts

+export type DeduplicationScope = 'node' | 'workflow';
+export type DeduplicationItemTypes = string | number;
+export type DeduplicationMode = 'entries' | 'latestIncrementalKey' | 'latestDate';


Thank you, more descriptive now 👌

tomi · 2024-10-09T16:52:24Z

packages/cli/src/deduplication/deduplication-helper.ts

+		this.validateMode(processedData, options);
+
+		if (['latestIncrementalKey', 'latestDate'].includes(options.mode)) {
+			const incomingItems = DeduplicationHelper.sortEntries(items, options.mode);


This method is quite long and seems to consists of two branches. WDYT about moving the branches to their own methods? I.e. something like this:

if (options.mode === 'latestIncrementalKey' || options.mode === 'latestDate') { return await this.dedupeAndRecordByLatest(...) } else { return await this.dedupeAndRecordByEntries(...) }

cypress · 2024-10-09T17:05:39Z

n8n Run #7316

Run Properties: Passed #7316 • c29aecbc15: 🌳 🖥️ browsers:node18.12.0-chrome107 🤖 ShireenMissi 🗃️ e2e/*

Project	`n8n`
Run status	`Passed #7316`
Run duration	`54m 04s`
Commit	`c29aecbc15: 🌳 🖥️ browsers:node18.12.0-chrome107 🤖 ShireenMissi 🗃️ e2e/*`
Committer	`Shireen Missi`
View all properties for this run ↗︎

Test results
Failures	`0`
Flaky	`0`
Pending	`0`
Skipped	`0`
Passing	`438`

github-actions · 2024-10-09T17:05:51Z

✅ All Cypress E2E specs passed

Co-authored-by: Tomi Turtiainen <10324676+tomi@users.noreply.github.com>

tomi

LGTM 🚀

packages/cli/src/deduplication/deduplication-helper.ts

github-actions · 2024-10-10T12:12:20Z

⚠️ Some Cypress E2E specs are failing, please fix them before merging

Co-authored-by: Tomi Turtiainen <10324676+tomi@users.noreply.github.com>

github-actions · 2024-10-10T13:07:50Z

⚠️ Some Cypress E2E specs are failing, please fix them before merging

github-actions · 2024-10-10T14:58:39Z

✅ All Cypress E2E specs passed

janober · 2024-10-16T11:43:02Z

Got released with n8n@1.64.0

ShireenMissi requested a review from a team as a code owner July 18, 2024 16:40

ShireenMissi marked this pull request as draft July 18, 2024 16:40

ShireenMissi force-pushed the node-1487 branch from 45665d8 to 4c2b739 Compare July 18, 2024 16:42

n8n-assistant bot added core Enhancement outside /nodes-base and /editor-ui n8n team Authored by the n8n team node/new Creation of an entirely new node labels Jul 18, 2024

ShireenMissi force-pushed the node-1487 branch 10 times, most recently from fb0a47d to 509ebed Compare July 24, 2024 16:17

ShireenMissi force-pushed the node-1487 branch from 509ebed to ce2690d Compare August 13, 2024 15:39

ShireenMissi force-pushed the node-1487 branch from ce2690d to 357f95e Compare August 22, 2024 09:36

ShireenMissi force-pushed the node-1487 branch from 3b024d1 to b1d9b74 Compare September 2, 2024 16:32

ShireenMissi force-pushed the node-1487 branch 6 times, most recently from 97f5711 to ca02cf9 Compare September 13, 2024 15:30

ShireenMissi marked this pull request as ready for review September 15, 2024 18:09

ShireenMissi removed the node/new Creation of an entirely new node label Sep 15, 2024

ShireenMissi force-pushed the node-1487 branch 3 times, most recently from 6b1d5b1 to 0ea7889 Compare September 19, 2024 07:51

Copy tweaks

2a1182c

ShireenMissi force-pushed the node-1487 branch from e82deb5 to 2a1182c Compare October 9, 2024 09:27

ShireenMissi added 2 commits October 9, 2024 10:41

support continue on fail

40845aa

refactor deduplication helper

e636b87

ShireenMissi changed the title ~~feat(core): Check data processed~~ feat(core): Dedupe Oct 9, 2024

ShireenMissi and others added 6 commits October 9, 2024 15:42

refactore and clean up unused parts

46e1445

Merge branch 'master' into node-1487

eeeeba2

clean up

b2e5155

Make workflow.id required, remove typecasts

2e2eb57

Use luxon based date parsing

4da4e9d

Improve checkProcessedAndRecord loop performance

c290e51

tomi previously approved these changes Oct 9, 2024

View reviewed changes

Update packages/nodes-base/nodes/Transform/RemoveDuplicates/utils.ts

2aa16d0

Co-authored-by: Tomi Turtiainen <10324676+tomi@users.noreply.github.com>

ShireenMissi dismissed tomi’s stale review via 2aa16d0 October 10, 2024 09:51

ShireenMissi added 2 commits October 10, 2024 12:19

addressed review comments

65dc5b1

fix formatting issue

2b6317d

tomi previously approved these changes Oct 10, 2024

View reviewed changes

packages/cli/src/deduplication/deduplication-helper.ts Outdated Show resolved Hide resolved

Update packages/cli/src/deduplication/deduplication-helper.ts

c29aecb

Co-authored-by: Tomi Turtiainen <10324676+tomi@users.noreply.github.com>

ShireenMissi dismissed tomi’s stale review via c29aecb October 10, 2024 12:50

tomi approved these changes Oct 10, 2024

View reviewed changes

ShireenMissi merged commit 52dd2c7 into master Oct 10, 2024
36 of 39 checks passed

ShireenMissi deleted the node-1487 branch October 10, 2024 15:12

github-actions bot mentioned this pull request Oct 16, 2024

🚀 Release 1.64.0 #11281

Merged

janober added the Released label Oct 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(core): Dedupe #10101

feat(core): Dedupe #10101

ShireenMissi commented Jul 18, 2024 •

edited

Loading

tomi left a comment

tomi Oct 9, 2024

ShireenMissi Oct 10, 2024

tomi Oct 9, 2024

ShireenMissi Oct 10, 2024

tomi Oct 9, 2024

tomi Oct 9, 2024

cypress bot commented Oct 9, 2024 •

edited

Loading

github-actions bot commented Oct 9, 2024

tomi left a comment

github-actions bot commented Oct 10, 2024

github-actions bot commented Oct 10, 2024

github-actions bot commented Oct 10, 2024

janober commented Oct 16, 2024

feat(core): Dedupe #10101

feat(core): Dedupe #10101

Conversation

ShireenMissi commented Jul 18, 2024 • edited Loading

Summary

Related Linear tickets, Github issues, and Community forum posts

Review / Merge checklist

tomi left a comment

Choose a reason for hiding this comment

tomi Oct 9, 2024

Choose a reason for hiding this comment

ShireenMissi Oct 10, 2024

Choose a reason for hiding this comment

tomi Oct 9, 2024

Choose a reason for hiding this comment

ShireenMissi Oct 10, 2024

Choose a reason for hiding this comment

tomi Oct 9, 2024

Choose a reason for hiding this comment

tomi Oct 9, 2024

Choose a reason for hiding this comment

cypress bot commented Oct 9, 2024 • edited Loading

n8n Run #7316

github-actions bot commented Oct 9, 2024

tomi left a comment

Choose a reason for hiding this comment

github-actions bot commented Oct 10, 2024

github-actions bot commented Oct 10, 2024

github-actions bot commented Oct 10, 2024

janober commented Oct 16, 2024

ShireenMissi commented Jul 18, 2024 •

edited

Loading

cypress bot commented Oct 9, 2024 •

edited

Loading