Heuristically/Optionally use sparse data structures for adjacency matrix aggregation

[flash1293]

Idea

The adjacency matrix aggregation currently has a default limit of 100 filters (index.max_adjacency_matrix_filters). This is a sensible value considering the quadratic memory complexity of the operation. For some use cases this is pretty restricting. Of course in these cases it is possible to increase this default level, but the memory pressure quickly becomes unfeasible.

I talked with Mark Harwood and we came to the conclusion that this limit could probably be increased if sparse data structures were used for storing intermediate results in cases of large matrices (> 100 filters), because the memory usage wouldn't grow as fast as it would with dense data structures. This would probably result in a space/time trade-off, but allow for new use cases.

API proposal

It's an option to not expose this in the API and apply a heuristic when to use which implementation (e.g. < 100 filters uses dense data structures, > 100 filters uses sparse data structures). The advantage of this is approach is that the user doesn't have to make a decision, but it could result in unexpected performance cliffs.

An alternative to that is to add a separate parameter to the request object (e.g. useSparseMatrix) to leave the decision to the user. In that case there should be a separate setting, e.g. index.max_sparse_adjacency_matrix_filters.

A cleaner way would be have a completely new aggregation sparse_adjacency_matrix to emphasize the distinction even more.

cc @markharwood

[elasticmachine]

Pinging @elastic/es-search

[elasticmachine]

Pinging @elastic/es-analytics-geo

[markharwood]

This is the code that pre-allocates an array of Bits objects for all possible sets (each of the filters and the intersections of all filters).
If the number of filters is N we will always need at least N equivalents of the Bits objects to represent each of the matching doc streams that need to be intersected. That can't be minimised.

The other pre-allocated Bits objects are lightweight BitsIntersector objects and we pre-allocate ((N*N) - N)/2 of these with all the possible pairings of the Bits objects used in our collection of N filters.
These are arguably avoidable - they make the code simpler but the logic they encapsulate could be extracted into loops that iterate over the N filters.

A motivation for capping the number of filters is not driven by the cost of these BitsIntersector objects - it will be

1. The number of independent filters objects, (for the same reason Lucene Boolean queries typically set a limit of 1024 clauses to prevent excessive disk seeks).
2. The number of buckets created in RAM (especially when used with child or parent aggregations).