This repository has been archived by the owner on Apr 4, 2023. It is now read-only.
Fix bug 2945/3021 (missing key in documents database) #734
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
loiclec
added
bug
loiclec
force-pushed
the
bug-3021-test
branch
from
e44ca84 to
2460258
Compare
loiclec
changed the title
loiclec
force-pushed
the
bug-3021-test
branch
from
2460258 to
09fb37b
Compare
If the document import replaces a document using hard deletion
loiclec
force-pushed
the
bug-3021-test
branch
from
09fb37b to
e3ee553
Compare
loiclec
changed the title
Kerollmops
suggested changes
Dec 12, 2022
irevoire
approved these changes
Dec 13, 2022
Kerollmops
approved these changes
Dec 13, 2022
|
Build succeeded:
|
bors bot
added a commit
that referenced
this pull request
Dec 14, 2022
739: Cherry pick bug fixes for v0.37.3 r=curquiza a=curquiza Integrate #734 and #737 into the `release-v0.37.3` branch to avoid to release on `main` <s>Wait for meilisearch/meilisearch#3235 investigation without merging this PR. Indeed, if a bug fix on milli's side, we might wait for it before merging this PR/</s> -> no need to wait, not a bug Co-authored-by: ManyTheFish <many@meilisearch.com> Co-authored-by: Loïc Lecrenier <loic.lecrenier@me.com>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request
Related issue
Fixes (partially, until merged into meilisearch) meilisearch/meilisearch#2945 (until we integrate the new milli bump into meilisearch).
Note that a dump will not be sufficient to upgrade from meilisearch v0.30.2 to meilisearch v0.30.3 due to this fix because the bug could have caused the
documentsdatabase to be corrupted. Instead, a full manual reimport of the documents will be necessary.What does this PR do?
There was a bug happening when:
The
IndexDocuments::executemethod would then perform the hard-deletion but forget to change theexternal_document_idsstructure appropriately. As a result, theexternal_document_idswould contain keys corresponding to documents that do no exist anymore.To fix this bug, I split the
DeleteDocuments::executemethod into two:execute_innerandexecute.execute_innerreturns aDetailedDocumentDeletionResultwhich says whether soft-deletion was used or notexecutekeeps the exact same signature and behaviourThen, when deleting replaced documents inside
IndexDocuments::execute, we callDeleteDocuments::execute_innerinstead ofDeleteDocuments::execute. If soft-deletion was used, nothing more is done. But if hard-deletion was used, we remove every reference to soft-deleted documents in the newexternal_documents_idsstructure.Correctness
update-fuzz-test), I created a fuzz-test that reproduces the past two bugs. This fuzz test cannot find this bug through any combination of some hand-selectedDocumentAddition / DocumentDeletion / DocumentClear / SettingsUpdateoperations. In that test, each relevant operations can be executed with or without soft-deletion, and document additions can be done in batches, replacing or updating existing documents.