Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(core): Check data processed #5178

Draft
wants to merge 32 commits into
base: master
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
95626c8
feat(core): Make it possible to check if data got already processed
janober Jul 17, 2022
c18304c
:zap: Some cleanup
janober Jul 17, 2022
b708890
:zap: Ensure the workflow has an ID and that certain errors are thrown
janober Jul 17, 2022
0eaa73c
:heavy_check_mark: Add basic tests
janober Jul 17, 2022
8b52f64
:twisted_rightwards_arrows: Merge branch 'master' into check-data-pro…
janober Jul 23, 2022
99496d5
:zap: Make it easy to check if items got already processed and record
janober Jul 23, 2022
7af344e
:twisted_rightwards_arrows: Merge branch 'master' into check-data-pro…
janober Jul 29, 2022
82a84a4
:heavy_check_mark: Add tests that different contexts are independent of
janober Jul 30, 2022
6ffea71
:zap: Clean up data in database on workflow update & delete
janober Jul 30, 2022
d173dde
:twisted_rightwards_arrows: Merge branch 'master' into check-data-pro…
janober Aug 6, 2022
cbf3238
:zap: Fix test to make work with new node ID
janober Aug 7, 2022
900525f
:zap: Small simplification
janober Aug 7, 2022
2783c1c
:zap: Minor improvements
janober Aug 7, 2022
dc1c82f
:zap: Init database depending on loaded module
janober Aug 7, 2022
b2dfb63
:shirt: Fix lint issue
janober Aug 7, 2022
4478c72
:twisted_rightwards_arrows: Merge branch 'master' into check-data-pro…
janober Sep 27, 2022
f280617
:zap: Fix typo
janober Sep 27, 2022
270ea5c
Merge branch 'master' into check-data-processed
janober Oct 22, 2022
93376f5
:zap: Change to single entry
janober Oct 22, 2022
4c3c5b2
:zap: Add support for maxEntries
janober Oct 22, 2022
d91fb1b
:zap: Add support for mode "latest"
janober Oct 24, 2022
7f01af3
:zap: Minor improvements
janober Oct 25, 2022
bfd7fce
Merge branch 'master' into check-data-processed
janober Dec 7, 2022
f4987fe
Merge branch 'master' into check-data-processed
janober Dec 7, 2022
73bb17f
Merge branch 'master' into check-data-processed
janober May 7, 2023
3ff37f8
:zap: Add missing file
janober May 12, 2023
fb4d700
Merge branch 'master' into check-data-processed
janober May 12, 2023
bb46814
Merge branch 'master' into check-data-processed
janober May 31, 2023
9a39ab8
Merge branch 'master' into check-data-processed
janober Jul 15, 2023
00d1c17
Merge branch 'master' into check-data-processed
janober Jul 27, 2023
4d195a9
:zap: Fix issues after merge
janober Jul 27, 2023
ca93689
Merge branch 'check-data-processed' of github.com:n8n-io/n8n into che…
janober Jul 27, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
⚡ Ensure the workflow has an ID and that certain errors are thrown
janober committed Jul 17, 2022

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
commit b7088908ce91ef89f8eb8326237e46ac1604023c
15 changes: 13 additions & 2 deletions packages/cli/src/ProcessedDataManagers/NativeDatabase.ts
Original file line number Diff line number Diff line change
@@ -79,10 +79,13 @@ export class ProcessedDataManagerNativeDatabase implements IProcessedDataManager
items: string[],
context: IProcessedDataContext,
contextData: ICheckProcessedContextData,
// maxRecords: number,
): Promise<ICheckProcessedOutput> {
const dbContext = ProcessedDataManagerNativeDatabase.createContext(context, contextData);

if (!contextData.workflow.id) {
throw new Error('Workflow has to have an ID set!');
}

if (ProcessedDataManagerNativeDatabase.dbType === 'sqlite') {
// SQLite is used as database
// In SQLite we sadly have to check each of the items one by one instead of in bulk.
@@ -94,7 +97,7 @@ export class ProcessedDataManagerNativeDatabase implements IProcessedDataManager
for (const item of items) {
upsertPromises.push(
Db.collections.ProcessedData.insert({
workflowId: contextData.workflow.id as string,
workflowId: contextData.workflow.id,
context: dbContext,
value: ProcessedDataManagerNativeDatabase.createValueHash(item),
}),
@@ -108,6 +111,14 @@ export class ProcessedDataManagerNativeDatabase implements IProcessedDataManager
promiseResults.forEach((result) => {
item = items.shift() as string;
if (result.status === 'rejected') {
if (result.reason.code !== 'SQLITE_CONSTRAINT') {
// We expect constraint issues, as that means that they value got already processed
// before but all other ones are actual errors which have to be thrown.

// eslint-disable-next-line @typescript-eslint/restrict-template-expressions
throw new Error(`Problem inserting ProcessedData: '${result.reason.toString()}'`);
}

// Got already processed
processedItems.push(item);
} else {