Composite aggregates #3266

normen662 · 2025-03-21T16:44:17Z

This PR adds the ability to plan arbitrary aggregation queries using aggregation indexes as well as intersections of aggregation indexes. It also corrects/adds the infrastructure to deal with complicated order-by requirements.

refactor data access rules into a data access rule for value indexes and one for aggregation indexes
have both of these data access rule implementations share the maximum code (only intersection planning is different)
intersection planning for aggregation indexes
- because of the same reasoning as regular index intersections AND in order to answer the query as these indexes carry data that is not part of the underlying table (the aggregation)
- rollups of finer-granularity index scans if such a roll-up can be useful in answering the query
order by requirements are treated in the same framework as all other kinds of expressions (using M3 maps, translations, etc.)

hatyo · 2025-07-31T13:50:14Z

...ore/src/main/java/com/apple/foundationdb/record/query/combinatorics/EnumeratingIterable.java

@@ -44,6 +44,10 @@ static <T> EnumeratingIterable<T> singleIterable(@Nonnull final T singleElement)
        return new SingleIterable<>(singleElement);
    }

+    static <T> EnumeratingIterable<T> emptyOnEmptyIterable() {
+        return new SingleIterable<>();


It looks like the reason for using SingleIterator (with null argument) instead of EmptyIterator is related to returning empty list, compared to returning null in EmptyIterator when calling computeNext, is that so, why not just use EmptyIterator instead and let the customer deal with the null?

hatyo · 2025-07-31T14:26:57Z

...java/com/apple/foundationdb/record/query/plan/cascades/expressions/RelationalExpression.java

     * @return {@code Optional.empty()} if the match could not be adjusted, Optional.of(matchInfo) for a new adjusted
     *         match, otherwise.
     */
    @Nonnull
-    default Optional<MatchInfo> adjustMatch(@Nonnull final PartialMatch partialMatch) {
+    default Optional<MatchInfo> adjustMatch(@Nonnull final PartialMatch partialMatch,
+                                            @Nonnull final Quantifier candidateQuantifier) {


Isn't this quantifier the one of the child? can we not always infer it considering that this should have a single child.

hatyo · 2025-07-31T14:30:42Z

...a/com/apple/foundationdb/record/query/plan/cascades/expressions/MatchableSortExpression.java

@@ -212,7 +212,8 @@ public MatchableSortExpression translateCorrelations(@Nonnull final TranslationM

    @Nonnull
    @Override
-    public Optional<MatchInfo> adjustMatch(@Nonnull final PartialMatch partialMatch) {
+    public Optional<MatchInfo> adjustMatch(@Nonnull final PartialMatch partialMatch,
+                                           @Nonnull final Quantifier candidateQuantifier) {
        final var childMatchInfo = partialMatch.getMatchInfo();
        final var maxMatchMap = childMatchInfo.getMaxMatchMap();
        final var innerQuantifier = Iterables.getOnlyElement(getQuantifiers());


innerQuantifier and candidateQuantifier are always the same I think.

hatyo · 2025-07-31T14:35:02Z

yaml-tests/src/main/java/com/apple/foundationdb/relational/yamltests/YamlTestExtension.java

@@ -70,7 +70,7 @@ public class YamlTestExtension implements TestTemplateInvocationContextProvider,
    private final boolean includeMethodInDescriptions;

    public YamlTestExtension() {
-        this(null, false);
+        this(null, true);


is this necessary?

hatyo · 2025-07-31T14:36:19Z

yaml-tests/src/test/resources/bitmap-aggregate-index.yamsql

-      - unorderedResult: [{BITMAP: xStartsWith_1250'0200004', 'CATEGORY': 'hello', 'OFFSET':0},
-                          {BITMAP: xStartsWith_1250'02', 'CATEGORY': 'hello', 'OFFSET':10000},
-                          {BITMAP: xStartsWith_1250'0400008', 'CATEGORY': 'world', 'OFFSET':0}]
+    #-


why are these tests commented out?

hatyo · 2025-08-04T11:44:06Z

...in/java/com/apple/foundationdb/record/query/plan/cascades/rules/AggregateDataAccessRule.java

+            return IntersectionResult.noViableIntersection();
+        }
+
+        final var compensation =


I believe we can immediately return IntersectionResult.noViableIntersection() if the resulting intersection compensation is impossible, correct?

hatyo · 2025-08-04T11:52:55Z

...in/java/com/apple/foundationdb/record/query/plan/cascades/rules/AggregateDataAccessRule.java

+        return RecordConstructorValue.ofColumns(columnBuilder.build());
+    }
+
+    private static TranslationMap computeTranslationMap(@Nonnull final CorrelationIdentifier intersectionAlias,


Can be tagged with @Nonnull

hatyo · 2025-08-04T12:48:02Z

...m/apple/foundationdb/record/query/plan/cascades/expressions/LogicalTypeFilterExpression.java

            }
+            isCompensationImpossible |= resultCompensationFunction.isImpossible();
+
+            groupByMappings =


It would be nice to add some comments explaining the handling of matched group by aggregations and groups here.

hatyo · 2025-08-04T12:51:54Z

...com/apple/foundationdb/record/query/plan/plans/RecordQueryMultiIntersectionOnValuesPlan.java

+                                .map(childPlan -> (Function<byte[], RecordCursor<QueryResult>>)
+                                        ((byte[] childContinuation) -> childPlan
+                                                .executePlan(store, context, childContinuation, childExecuteProperties)))
+                                .collect(Collectors.toList()),


Can we use ImmutableList here instead?

hatyo · 2025-08-04T12:54:11Z

...in/java/com/apple/foundationdb/record/query/plan/cascades/rules/AggregateDataAccessRule.java

+            }
+        }
+
+        return IntersectionResult.of(hasCommonOrdering ? intersectionOrdering : null, compensation,


I don't think it is possible to retrieve an IntersectionResult which non-empty list of RelationalExpressions and null-common ordering, if we examine the code flow above.

Having said that, we could probably cleanup the code, such that we can remove the hasCommonFlag maybe?

normen662 force-pushed the composite-aggregates branch from a25ae95 to 4c9ec49 Compare March 24, 2025 08:24

normen662 force-pushed the composite-aggregates branch 2 times, most recently from 72f7a46 to 2f5e9b8 Compare April 2, 2025 19:26

normen662 force-pushed the composite-aggregates branch from 8f97639 to 55cd005 Compare April 10, 2025 12:47

normen662 requested review from hatyo, alecgrieser and MMcM April 15, 2025 08:20

normen662 added the enhancement New feature or request label Apr 15, 2025

normen662 force-pushed the composite-aggregates branch 2 times, most recently from adec3d2 to 91ed1cf Compare May 27, 2025 11:02

normen662 removed request for MMcM, alecgrieser and hatyo May 27, 2025 11:02

normen662 force-pushed the composite-aggregates branch from 91ed1cf to 6fa2c04 Compare July 9, 2025 16:48

normen662 added 16 commits July 9, 2025 17:49

save point

183d774

pulling up matchedAggregationValueMap works

1779ec5

pre pulling up unmatched values

4c2603f

pullng up unmatched values

6e98b4c

intersection complete

3dcf180

intersection complete

72c8f9e

save-point

3c0d29e

serialization for multi intersection implemented

6cb4d89

first plans

59ae202

basic aggregation queries using agg indexes intersections work

9458645

more agg queriers work

c68b0e0

rebased; regular intersections work; pre rollup

db8858d

roll ups work

2ea1a71

rollups work

6adbd65

ordering sort of works

9059271

agg testcases run

02d28f6

normen662 added 7 commits July 9, 2025 17:49

cleaning up

a7b022d

build somewhat stable

f8c69a7

some polishing

5b0b266

updating tests and writing documentation

42baaef

fixups after rebasing

5fde97b

regenerated metrics files

e428f08

addressing rebase fallout

0aeab9c

normen662 force-pushed the composite-aggregates branch from 6fa2c04 to 0aeab9c Compare July 9, 2025 16:52

hatyo self-requested a review July 31, 2025 13:16

hatyo requested changes Aug 4, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Composite aggregates #3266

Composite aggregates #3266

Uh oh!

normen662 commented Mar 21, 2025 •

edited

Loading

Uh oh!

hatyo Jul 31, 2025

Uh oh!

hatyo Jul 31, 2025

Uh oh!

hatyo Jul 31, 2025

Uh oh!

hatyo Jul 31, 2025

Uh oh!

hatyo Jul 31, 2025

Uh oh!

hatyo Aug 4, 2025

Uh oh!

hatyo Aug 4, 2025

Uh oh!

hatyo Aug 4, 2025

Uh oh!

hatyo Aug 4, 2025

Uh oh!

hatyo Aug 4, 2025

Uh oh!

Uh oh!

Composite aggregates #3266

Are you sure you want to change the base?

Composite aggregates #3266

Uh oh!

Conversation

normen662 commented Mar 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

normen662 commented Mar 21, 2025 •

edited

Loading