Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

smarter nested column index utilization #13977

Merged

Commits on Mar 25, 2023

  1. smarter nested column index utilization

    changes:
    * adds skipValueRangeIndexScale and skipValuePredicateIndexScale to ColumnConfig (e.g. DruidProcessingConfig) available as system config via druid.processing.indexes.skipValueRangeIndexScale and druid.processing.indexes.skipValuePredicateIndexScale
    * NestedColumnIndexSupplier uses skipValueRangeIndexScale and skipValuePredicateIndexScale to multiply by the total number of rows to be processed to determine the threshold at which we should no longer consider using bitmap indexes because it will be too many operations
    * Default values for skipValueRangeIndexScale and skipValuePredicateIndexScale have been initially set to 0.08, but are separate to allow independent tuning
    * these are not documented on purpose yet because they are kind of hard to explain, the mainly exist to help conduct larger scale experiments than the jmh benchmarks used to derive the initial set of values
    * these changes provide a pretty sweet performance boost for filter processing on nested columns
    clintropolis committed Mar 25, 2023
    Configuration menu
    Copy the full SHA
    d9cfb7c View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    b51445a View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    59980cb View commit details
    Browse the repository at this point in the history
  4. fix tests

    clintropolis committed Mar 25, 2023
    Configuration menu
    Copy the full SHA
    4402479 View commit details
    Browse the repository at this point in the history

Commits on Apr 5, 2023

  1. Configuration menu
    Copy the full SHA
    4627125 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    862a5dd View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    df0ec73 View commit details
    Browse the repository at this point in the history
  4. oops

    clintropolis committed Apr 5, 2023
    Configuration menu
    Copy the full SHA
    3bd7d2d View commit details
    Browse the repository at this point in the history
  5. refactor

    clintropolis committed Apr 5, 2023
    Configuration menu
    Copy the full SHA
    131b5a1 View commit details
    Browse the repository at this point in the history
  6. fix inspection

    clintropolis committed Apr 5, 2023
    Configuration menu
    Copy the full SHA
    9b3170e View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    ed3fc6c View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    413e5dd View commit details
    Browse the repository at this point in the history

Commits on Apr 6, 2023

  1. Configuration menu
    Copy the full SHA
    0d36125 View commit details
    Browse the repository at this point in the history
  2. Merge remote-tracking branch 'origin/smarter-nested-column-indexes' i…

    …nto smarter-nested-column-indexes
    clintropolis committed Apr 6, 2023
    Configuration menu
    Copy the full SHA
    afa6a05 View commit details
    Browse the repository at this point in the history