Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable sort optimization for all NumericTypes #6420

Closed
wants to merge 9 commits into from

Conversation

gashutos
Copy link
Contributor

@gashutos gashutos commented Feb 21, 2023

Description

Enable sort optimization for all remaining Numeric types which were not covered in our last PR 6321
The main issue supporting these remaining NumericTypes was its merging capabilities during field mapping change for indices. That is resolved in this PR by introducing wider size comparator during merge.

Below remainig types were enabled as a part of this PR.

  1. Boolean
  2. Bytes
  3. Short
  4. Int
  5. Half_float (it always uses custom comparator though)
  6. Float

The performance gain we had with this is very huge, almost 10x gain on server side latency comparison. I have listed down few benchmarks in testing section.

Performance improvement

Below is server side latency comparison. Units are in ms.
Dataset as of now I have are smaller for float & short types, but improvement in 'x' wise is pretty wide.

NumericType New sort latency Old sort latency
SHORT 13 130
INT 150 1411
FLOAT 15 240

Issues Resolved

6326

Tested

  • Added extensive unit tests to check if merge works properly in case indices have different numeric types sorted.
  • Tested creating managed service domain and did performance testing, below were the results.
  • Reindexed indices with different mapping and checked things are working fine with optimization gains.

Signed-off-by: gashutos <gashutos@amazon.com>
Signed-off-by: gashutos <gashutos@amazon.com>
Signed-off-by: gashutos <gashutos@amazon.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@gashutos
Copy link
Contributor Author

Fixing integ tests..

@gashutos gashutos deleted the branch opensearch-project:main February 21, 2023 22:39
@gashutos gashutos closed this Feb 21, 2023
@gashutos gashutos deleted the main branch February 21, 2023 22:39
@gashutos
Copy link
Contributor Author

Closed this due to branch mess up :(
Created newer one #6424

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant