Fix TanStack DB groupBy duplicate insert errors#1030
Closed
KyleAMathews wants to merge 6 commits intomainfrom
Closed
Fix TanStack DB groupBy duplicate insert errors#1030KyleAMathews wants to merge 6 commits intomainfrom
KyleAMathews wants to merge 6 commits intomainfrom
Conversation
When processing incremental updates to groupBy queries, the D2 pipeline might emit an insert for an updated aggregate without a corresponding delete in certain edge cases. This caused "already exists" errors in the live query collection. Added a defensive check in applyChanges() that treats inserts as updates when the key already exists, but only for queries without custom getKey (like groupBy). Queries with custom getKey + joins still throw errors as expected to alert users of potential data issues. Added comprehensive test suite for groupBy incremental update scenarios: - Basic incremental updates with same groupBy key - Multiple groups being updated - Sum aggregate with incremental updates - Batch vs incremental processing - Subquery pattern (groupBy used as source for another query) - Rapid sequential inserts - Multiple events in single batch
|
More templates
@tanstack/angular-db
@tanstack/db
@tanstack/db-ivm
@tanstack/electric-db-collection
@tanstack/offline-transactions
@tanstack/powersync-db-collection
@tanstack/query-db-collection
@tanstack/react-db
@tanstack/rxdb-db-collection
@tanstack/solid-db
@tanstack/svelte-db
@tanstack/trailbase-db-collection
@tanstack/vue-db
commit: |
Contributor
|
Size Change: +35 B (+0.04%) Total Size: 89.5 kB
ℹ️ View Unchanged
|
Contributor
|
Size Change: 0 B Total Size: 3.35 kB ℹ️ View Unchanged
|
Update @tanstack/query-db-collection from ^1.0.7 to ^1.0.8 in: - examples/react/todo - examples/solid/todo This fixes the sherif check that requires consistent dependency versions across the workspace.
The previous fix in applyChanges was insufficient because the actual error was thrown in the sync layer. This adds a fallback that converts inserts to updates for live queries without custom getKey (like groupBy) when the key already exists, preventing duplicate key errors during incremental updates.
Added a test that directly exercises the sync layer fix by: 1. Setting up a collection with utils[LIVE_QUERY_INTERNAL].hasCustomGetKey = false (simulating a live query without custom getKey, like groupBy) 2. Inserting a document with a specific key 3. Attempting to insert another document with the same key (without delete) Without the fix, this would throw "Cannot insert document with key ... already exists". With the fix, the insert is converted to an update. Also added a test to verify regular collections still throw errors on duplicate inserts.
…plicate-inserts-PvGLM # Conflicts: # examples/react/todo/package.json # examples/solid/todo/package.json # pnpm-lock.yaml
Collaborator
Author
|
#1044 also fixes this! So closing |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When processing incremental updates to groupBy queries, the D2 pipeline might emit an insert for an updated aggregate without a corresponding delete in certain edge cases. This caused "already exists" errors in the live query collection.
Added a defensive check in applyChanges() that treats inserts as updates when the key already exists, but only for queries without custom getKey (like groupBy). Queries with custom getKey + joins still throw errors as expected to alert users of potential data issues.
Added comprehensive test suite for groupBy incremental update scenarios:
🎯 Changes
✅ Checklist
pnpm test:pr.🚀 Release Impact