Incorporate `estimatedComputeCost` into all `BitmapColumnIndex` classes. #17125

cecemei · 2024-09-20T19:32:08Z

Incorporate estimatedComputeCost into all BitmapColumnIndex classes.

Description

In #17055, we added estimatedIndexComputeCost field to FilterBundle.Builder class, which would be used to sort child filters in AndFilter/OrFilter. The goal is to compute less expensive filters first, thereby enhancing query performance. This PR aims to incorporate estimatedComputeCost into all BitmapColumnIndex classes, serving as an initial measure for estimating the cost of filters.

An overall approach of estimating the cost is to assess how many bitmaps we expect to union or intersect. Dictionary lookup would also incur some overhead.

AllTrueBitmapColumnIndex, AllFalseBitmapColumnIndex, AllUnknownBitmapColumnIndex. The cost is 0.
SimpleImmutableBitmapIndex. The cost is 0.
SimpleBitmapColumnIndex instances. It generally involves one binary search, and maybe union with null bitmap. The cost is 1.
- Note I changed one usage in ListFilteredDruidPredicateIndexes to use DictionaryScanningBitmapIndex instead.
DictionaryRangeScanningBitmapIndex. The cost is the size of scanning range.
DictionaryScanningBitmapIndex. The cost is the size of dictionary.
BaseValueSetIndexesFromIterable.
- buildBitmapColumnIndexFromSortedIteratorScan. Cost is max of value set size and dictionary size.
- buildBitmapColumnIndexFromSortedIteratorBinarySearch. Cost is value set size.
- buildBitmapColumnIndexFromIteratorBinarySearch. Cost is value set size.
NestedVariantStringValueSetIndexes. For each value, we need to look up from three global dictionaries (stringDictionary, longDictionary and doubleDictionary), and one local dictionary. Therefore we define a base cost of 3 (INDEX_COMPUTE_SCALE). The cost of building index from value set would be 3 * size of value set.
IsBooleanFilter and NotFilter, the cost is the same as baseIndex.
AndFilter and OrFilter, cost is 0 since the bundle would sum up the cost of its child filters.
SpatialFilter. There's no good way to define cost, so putting down max integer for now, it'll always be evaluated last.

Turned on CURSOR_AUTO_ARRANGE_FILTERS by default in this PR.

While working on this PR, I had some ideas on refactoring some of the usages of BitmapColumnIndex, specifically:

Consolidate the usage of SimpleImmutableBitmapIndex and SimpleBitmapColumnIndex. The BitmapColumnIndex interface defines a method computeBitmapResult(BitmapResultFactory<T> bitmapResultFactory, boolean includeUnknown). The includeUnknown param is not used in SimpleImmutableBitmapIndex because the bitmap is already pre-computed. Maybe we could have an UnknownBitmapSupplier instead.
Create a DictionaryBinarySearchBitmapIndex class which could extract the iterator definition out, maybe can replace all usages of SimpleImmutableBitmapDelegatingIterableIndex.

Benchmark comparison

I'm comparing the results with CURSOR_AUTO_ARRANGE_FILTERS flag enabled and disabled.

query 1

SELECT string2, SUM(long1) FROM foo WHERE string5 LIKE '%1%' AND string1 = '1000' GROUP BY 1 ORDER BY 2

Flag off

Benchmark                        (deferExpressionDimensions)  (query)  (rowsPerSegment)  (schema)  (vectorize)  Mode  Cnt    Score    Error  Units
SqlExpressionBenchmark.querySql         fixedWidthNonNumeric       46           5000000  explicit        force  avgt    5  326.808 ±  5.295  ms/op
SqlExpressionBenchmark.querySql         fixedWidthNonNumeric       46           5000000      auto        force  avgt    5  313.831 ±  3.502  ms/op

Flag on

Benchmark                        (deferExpressionDimensions)  (query)  (rowsPerSegment)  (schema)  (vectorize)  Mode  Cnt    Score    Error  Units
SqlExpressionBenchmark.querySql         fixedWidthNonNumeric       46           5000000  explicit        force  avgt    5  110.170 ±  4.968  ms/op
SqlExpressionBenchmark.querySql         fixedWidthNonNumeric       46           5000000      auto        force  avgt    5  109.375 ±  1.388  ms/op

query 2

SELECT string2, SUM(long1) FROM foo WHERE string5 LIKE '%1%' AND (string3 in ('1', '10', '20', '22', '32') AND long2 IN (1, 19, 21, 23, 25, 26, 46) AND double3 < 1010.0 AND double3 > 1000.0 AND (string4 = '1' OR REGEXP_EXTRACT(string1, '^1') IS NOT NULL OR REGEXP_EXTRACT('Z' || string2, '^Z2') IS NOT NULL)) AND string1 = '1000' GROUP BY 1 ORDER BY 2

Flag off

Benchmark                        (deferExpressionDimensions)  (query)  (rowsPerSegment)  (schema)  (vectorize)  Mode  Cnt    Score    Error  Units
SqlExpressionBenchmark.querySql         fixedWidthNonNumeric       47           5000000  explicit        force  avgt    5  333.207 ± 14.524  ms/op
SqlExpressionBenchmark.querySql         fixedWidthNonNumeric       47           5000000      auto        force  avgt    5  334.476 ± 39.445  ms/op

Flag on

Benchmark                        (deferExpressionDimensions)  (query)  (rowsPerSegment)  (schema)  (vectorize)  Mode  Cnt    Score    Error  Units
SqlExpressionBenchmark.querySql         fixedWidthNonNumeric       47           5000000  explicit        force  avgt    5  106.400 ± 14.760  ms/op
SqlExpressionBenchmark.querySql         fixedWidthNonNumeric       47           5000000      auto        force  avgt    5  105.784 ± 26.924  ms/op

query 3

SELECT SUM(long1) FROM foo WHERE string5 LIKE '%1%' AND string1 = '1000'

Flag off

Benchmark                        (query)  (rowsPerSegment)  (schema)  (stringEncoding)  (vectorize)  Mode  Cnt    Score   Error  Units
SqlNestedDataBenchmark.querySql       56           5000000  explicit              none        force  avgt    5  195.104 ± 3.935  ms/op
SqlNestedDataBenchmark.querySql       56           5000000      auto              none        force  avgt    5  224.047 ± 1.598  ms/op

Flag on

Benchmark                        (query)  (rowsPerSegment)  (schema)  (stringEncoding)  (vectorize)  Mode  Cnt    Score   Error  Units
SqlNestedDataBenchmark.querySql       56           5000000  explicit              none        force  avgt    5  8.200 ± 0.473  ms/op
SqlNestedDataBenchmark.querySql       56           5000000      auto              none        force  avgt    5  8.243 ± 0.809  ms/op

This PR has:

processing/src/main/java/org/apache/druid/segment/virtual/ListFilteredVirtualColumn.java

+      return ListFilteredDimensionSpec.filterAllowList(
+          values,
+          factory.makeDimensionSelector(delegate),
+          delegate.getExtractionFn() != null


processing/src/main/java/org/apache/druid/segment/virtual/ListFilteredVirtualColumn.java

+      return ListFilteredDimensionSpec.filterDenyList(
+          values,
+          factory.makeDimensionSelector(delegate),
+          delegate.getExtractionFn() != null


…1-clone

benchmarks/src/test/java/org/apache/druid/benchmark/query/SqlNestedDataBenchmark.java

processing/src/main/java/org/apache/druid/segment/index/IndexedUtf8ValueIndexes.java

processing/src/test/java/org/apache/druid/segment/filter/FilterBundleTest.java

clintropolis · 2024-09-23T19:44:40Z

processing/src/main/java/org/apache/druid/segment/index/SimpleImmutableBitmapIndex.java

+  @Override
+  public int estimatedComputeCost()
+  {
+    return 0;


it looks like this is mainly used for null value index, should this be 1 to be consistent with the equality indexes, like ValueIndexes.forValue, since the null indexes still have a bitmap?

Right this index seems mainly for null index, so it's just 1 bitmap with no union. When I looked up forValue seems like it's possible there're two bitmaps (one for the value and one for null) with one union. That's why i decided 0 for this, and 1 for other SimpleBitmapIndex instances. I feel SimpleImmutableBitmapIndex is slightly cheaper since no binary search for dictionary and no bitmap union.

clintropolis · 2024-09-23T20:06:12Z

processing/src/main/java/org/apache/druid/segment/nested/NestedFieldColumnIndexSupplier.java

@@ -1204,6 +1256,9 @@ private abstract class NestedVariantIndexes
    final FrontCodedIntArrayIndexed arrayDictionary = globalArrayDictionarySupplier == null
                                                      ? null
                                                      : globalArrayDictionarySupplier.get();
+    // For every single String value, we need to look up indexes from stringDictionary, longDictionary and
+    // doubleDictionary. Hence, the compute cost for one value is 3.
+    static final int INDEX_COMPUTE_SCALE = 3;


i actually think 1 would probably be ok here too since we still only use a single bitmap, but this is also fine

cecemei · 2024-09-23T22:48:20Z

benchmarks/src/test/java/org/apache/druid/benchmark/query/SqlNestedDataBenchmark.java

@@ -199,7 +199,7 @@ public String getFormatString()
      // 42, 43 big cardinality like predicate filter
      "SELECT SUM(long1) FROM foo WHERE string5 LIKE '%1%'",
      "SELECT SUM(JSON_VALUE(nested, '$.long1' RETURNING BIGINT)) FROM foo WHERE JSON_VALUE(nested, '$.nesteder.string5') LIKE '%1%'",
-      // 44, 45 big cardinality like filter + selector filter
+      // 44, 45 big cardinality like filter + selector filter with different ordering


Ran the tests for 44,45,46,47. The scores are very similar, since we've the ordering:

Benchmark (query) (rowsPerSegment) (schema) (stringEncoding) (vectorize) Mode Cnt Score Error Units SqlNestedDataBenchmark.querySql 44 5000000 explicit none force avgt 5 7.753 ± 0.380 ms/op SqlNestedDataBenchmark.querySql 44 5000000 auto none force avgt 5 8.023 ± 0.940 ms/op SqlNestedDataBenchmark.querySql 45 5000000 explicit none force avgt 5 7.976 ± 0.735 ms/op SqlNestedDataBenchmark.querySql 45 5000000 auto none force avgt 5 7.820 ± 0.863 ms/op SqlNestedDataBenchmark.querySql 46 5000000 explicit none force avgt 5 7.495 ± 0.279 ms/op SqlNestedDataBenchmark.querySql 46 5000000 auto none force avgt 5 7.861 ± 0.691 ms/op SqlNestedDataBenchmark.querySql 47 5000000 explicit none force avgt 5 7.577 ± 0.405 ms/op SqlNestedDataBenchmark.querySql 47 5000000 auto none force avgt 5 7.735 ± 0.491 ms/op

Comparing with upstream/master branch:

Benchmark (query) (rowsPerSegment) (schema) (stringEncoding) (vectorize) Mode Cnt Score Error Units SqlNestedDataBenchmark.querySql 44 5000000 explicit none force avgt 5 215.894 ± 2.712 ms/op SqlNestedDataBenchmark.querySql 44 5000000 auto none force avgt 5 208.718 ± 5.545 ms/op SqlNestedDataBenchmark.querySql 45 5000000 explicit none force avgt 5 221.829 ± 7.157 ms/op SqlNestedDataBenchmark.querySql 45 5000000 auto none force avgt 5 216.632 ± 2.712 ms/op SqlNestedDataBenchmark.querySql 46 5000000 explicit none force avgt 5 7.431 ± 0.286 ms/op SqlNestedDataBenchmark.querySql 46 5000000 auto none force avgt 5 7.396 ± 0.186 ms/op SqlNestedDataBenchmark.querySql 47 5000000 explicit none force avgt 5 7.487 ± 0.287 ms/op SqlNestedDataBenchmark.querySql 47 5000000 auto none force avgt 5 7.451 ± 0.203 ms/op

clintropolis

🤘 🚀

processing/src/main/java/org/apache/druid/segment/filter/BoundFilter.java

clintropolis · 2024-09-24T23:52:09Z

processing/src/main/java/org/apache/druid/segment/index/BitmapColumnIndex.java

-  {
-    return Integer.MAX_VALUE;
-  }
+  int estimatedComputeCost();


i know this isn't new in this PR, but i feel like maybe the javadoc should mention that the estimated cost should be related to the number of bitmap operations that need to be performed to compute the filter bitmap

added more explanation on this.

…Filter.java Co-authored-by: Clint Wylie <cjwylie@gmail.com>

…es. (apache#17125) changes: * filter index processing is now automatically ordered based on estimated 'cost', which is approximated based on how many expected bitmap operations are required to construct the bitmap used for the 'offset' * cursorAutoArrangeFilters context flag now defaults to true, but can be set to false to disable cost based filter index sorting

…es. (#17125) (#17172) changes: * filter index processing is now automatically ordered based on estimated 'cost', which is approximated based on how many expected bitmap operations are required to construct the bitmap used for the 'offset' * cursorAutoArrangeFilters context flag now defaults to true, but can be set to false to disable cost based filter index sorting

github-actions bot added the Area - Segment Format and Ser/De label Sep 20, 2024

Add estimatedComputeCost for all BitmapColumnIndex classes.

e2ffa6a

cecemei force-pushed the make-filter-bundle-1-clone branch from 0a2d6ac to e2ffa6a Compare September 20, 2024 19:33

github-advanced-security bot found potential problems Sep 20, 2024

View reviewed changes

cecemei force-pushed the make-filter-bundle-1-clone branch from ee92478 to 3ee3900 Compare September 21, 2024 01:49

Add unit tests and benchmark tests.

8eeb5e0

cecemei force-pushed the make-filter-bundle-1-clone branch from 3ee3900 to 8eeb5e0 Compare September 21, 2024 02:00

cecemei marked this pull request as ready for review September 23, 2024 16:58

Merge remote-tracking branch 'origin/master' into make-filter-bundle-…

ed9a48a

…1-clone

clintropolis reviewed Sep 23, 2024

View reviewed changes

cecemei added 2 commits September 23, 2024 15:15

Responding to comments.

9b2e9cc

small edits

944c770

cecemei commented Sep 23, 2024

View reviewed changes

clintropolis approved these changes Sep 24, 2024

View reviewed changes

cecemei and others added 2 commits September 25, 2024 11:29

expand javadoc on estimateComputeCost function

6fe388a

Update processing/src/main/java/org/apache/druid/segment/filter/Bound…

8f18b1c

…Filter.java Co-authored-by: Clint Wylie <cjwylie@gmail.com>

clintropolis merged commit a2b011c into apache:master Sep 26, 2024
90 checks passed

clintropolis added this to the 31.0.0 milestone Sep 26, 2024

cecemei mentioned this pull request Sep 26, 2024

[Backport] Incorporate estimatedComputeCost into all BitmapColumnIndex classes. #17172

Merged

clintropolis added Performance Release Notes labels Sep 27, 2024

cecemei mentioned this pull request Oct 8, 2024

Druid 31 release notes updates writer-jill/druid#76

Merged

9 tasks

kfaraz mentioned this pull request Oct 11, 2024

[DRAFT] 31.0.0 Release Notes #17332

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Incorporate `estimatedComputeCost` into all `BitmapColumnIndex` classes. #17125

Incorporate `estimatedComputeCost` into all `BitmapColumnIndex` classes. #17125

cecemei commented Sep 20, 2024 •

edited

Loading

clintropolis Sep 23, 2024

cecemei Sep 23, 2024

clintropolis Sep 23, 2024

cecemei Sep 23, 2024

cecemei Sep 23, 2024

clintropolis left a comment

clintropolis Sep 24, 2024

cecemei Sep 25, 2024 •

edited

Loading

Incorporate estimatedComputeCost into all BitmapColumnIndex classes. #17125

Incorporate estimatedComputeCost into all BitmapColumnIndex classes. #17125

Conversation

cecemei commented Sep 20, 2024 • edited Loading

Description

Benchmark comparison

query 1

query 2

query 3

This PR has:

clintropolis Sep 23, 2024

Choose a reason for hiding this comment

cecemei Sep 23, 2024

Choose a reason for hiding this comment

clintropolis Sep 23, 2024

Choose a reason for hiding this comment

cecemei Sep 23, 2024

Choose a reason for hiding this comment

cecemei Sep 23, 2024

Choose a reason for hiding this comment

clintropolis left a comment

Choose a reason for hiding this comment

clintropolis Sep 24, 2024

Choose a reason for hiding this comment

cecemei Sep 25, 2024 • edited Loading

Choose a reason for hiding this comment

Incorporate `estimatedComputeCost` into all `BitmapColumnIndex` classes. #17125

Incorporate `estimatedComputeCost` into all `BitmapColumnIndex` classes. #17125

cecemei commented Sep 20, 2024 •

edited

Loading

cecemei Sep 25, 2024 •

edited

Loading