Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Handle delete cases with star tree index #16381

Closed
bharath-techie opened this issue Oct 18, 2024 · 0 comments · Fixed by #16380
Closed

[BUG] Handle delete cases with star tree index #16381

bharath-techie opened this issue Oct 18, 2024 · 0 comments · Fixed by #16380
Assignees
Labels
bug Something isn't working Indexing:Performance

Comments

@bharath-techie
Copy link
Contributor

bharath-techie commented Oct 18, 2024

Describe the bug

  1. Refresh fails when delete documents are performed
"reason": {
                    "type": "refresh_failed_engine_exception",
                    "reason": "Refresh failed",
                    "index": "logs-241998",
                    "shard": "0",
                    "index_uuid": "d02mxLclQrmb65qHY8i6fg",
                    "caused_by": {
                        "type": "file_already_exists_exception",
                        "reason": "/Users/gbh/Documents/dev/OpenSearch/build/testclusters/runTask-0/data/nodes/0/indices/d02mxLclQrmb65qHY8i6fg/0/index/_6.cidvd"
                    }
                }
  1. Merge gives inaccurate results when there are deleted documents

Related component

Search:Performance

To Reproduce

Perform refresh and merge operations with deleted documents.

Expected behavior

  1. Refresh / Lucene flush should work with star tree index irrespective of deleted documents. In order to do this, fix is idenitfied to include SegmentSuffix as part of custom write state and read state used during star tree file formats
  2. For merge , there are few cases where SegmentDocValuesProducer is present as DocValuesProducer in a segment instead of the CodecDocValuesProducer.
    So one change is to pick DocValuesProducer in the DVP list where star tree index is present - each field is associated with only one of the DocValuesProducer - even SegmentDocValuesProducer has a map of FieldInfo to DocValuesProducer which confirms the same. So StarTreeValues composite field too will be present in only one of the DocValuesProducers

Also, star tree index will not account for deleted documents except for cases where complete segments are dropped during segment merge. [ In which case star tree associated with that segment is also dropped ]
During merge, out of all the segments which does have star tree , we will merge based on the values already present in star tree index - hence deletes will not be accounted.

This issue fix handles the bug around enforcing the above behavior.

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

  • OS: [e.g. iOS]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working Indexing:Performance
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant