Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add support for ColumnConfig skipValueRangeIndexScale and skipValueRangeIndexScale to traditional string columns #15551

Closed

Conversation

clintropolis
Copy link
Member

@clintropolis clintropolis commented Dec 13, 2023

Description

Adds the enhancements from #13977 to traditional string columns to improve performance of filter processing by allowing obviously expensive bitmap operations to be skipped. While I haven't run the benchmarks, I assume a similar improvement for string columns that are not created with the 'auto' indexer as were seen in #13977. Previously this performance enhancement was only available for 'auto' and 'json' columns.

This is related to #15550 to help replace the 'auto' search strategy.

I've not documented the configs behind this yet in this PR because I'm still thinking on how to frame what these configs do in an understandable manner without requiring deep knowledge of how query processing actually works. For the most part these settings should probably not be tweaked by users, but will try to think of something and add docs in a follow-up PR.

Release note

String columns created with the 'string' column indexer now have an enhancement for filter match processing that was previously only available to columns created by the 'auto' indexer, and will automatically skip obviously expensive index computation for filters which would require a very large number of bitmap operations. This should improve performance, particularly when filter clauses contain a mix of simple and complex filters since it allows the complex filters. Previously Druid would always utilize indexes if they were available, and this behavior can be returned by setting druid.processing.skipValuePredicateIndexScale and druid.processing.skipValueRangeIndexScale to 1.0.


This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • been tested in a test Druid cluster.

@clintropolis
Copy link
Member Author

replaced by #15838

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant