Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix ElasticSearch error with filter deletion #584

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

antoinebhs
Copy link
Contributor

On a filter deletion with a lot of things to delete, ES returns a "All shards have failed" error.
That's because we are asking ElasticSearch to return all the equipment infos corresponding to deleted identifiables ids. If we have a lot of equipments deleted, the following list:
deletedEquipments.stream().map(EquipmentInfosToDelete::id).toList(),
can contain more than 10000 elements.
This equipment infos list is then used to avoid sending delete requests to ES for equipments that don't exist in the index. This is not necessary as ElasticSearch can handle this directly. In terms of performance, it's more efficient to send delete request with documents that don't exist instead of fetching the documents and deleting only the existing ones from my tests.

I added another constant for deletion partitioning as the existing one is too big for deletions. If there are too many ids in the request, the number of clauses is too large and can't be handled by ElasticSearch. The error looks like:

            "reason" : "maxClauseCount is set to 2114"

on my local deployment.
From the documentation, in the version we use there is a heuristic to determine the maxClauseCount from the available memory/heap (https://www.elastic.co/guide/en/elasticsearch/reference/current/search-settings.html).
I set it by default to 2048 to be below the limit in our local deployment but we could override this value in other environment if we're looking to improve performance of delete with a lot of values.

Signed-off-by: BOUHOURS Antoine <antoine.bouhours@rte-france.com>
@antoinebhs antoinebhs force-pushed the fix-delete-by-filter-es branch from e583e4d to a8f2521 Compare December 26, 2024 16:02
@antoinebhs
Copy link
Contributor Author

Looks like we still see deleted equipments in search

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant