Auto Indexing

[osopardo1]

Description

Adds new Auto Indexing functionality #244

Type of change

A new feature that enables indexing a DataFrame/Table without specifying the columnsToIndex.

The feature is not enabled by default. If you want to use it, you should add the necessary configuration.

spark.qbeast.index.columnsToIndex.auto=true
spark.qbeast.index.columnsToIndex.auto.max=10

Code

The idea is to have an ColumnsToIndexSelector interface in the core project with the following information:

trait ColumnsToIndexSelector[DATA] {

  /**
   * The maximum number of columns to index.
   * @return
   */
  def MAX_COLUMNS_TO_INDEX: Int

  /**
   * Selects the columns to index given a DataFrame
   * @param data
   *   the data to index
   * @return
   */
  def selectColumnsToIndex(data: DATA): Seq[String] =
    selectColumnsToIndex(data, MAX_COLUMNS_TO_INDEX)

  /**
   * Selects the columns to index with a given number of columns to index
   * @param data
   *   the data to index
   * @param numColumnsToIndex
   *   the number of columns to index
   * @return
   *   A sequence with the names of the columns to index
   */
  def selectColumnsToIndex(data: DATA, numColumnsToIndex: Int): Seq[String]

}

• MAX_COLUMNS_TO_INDEX: maximum number of columns to index for this implementation (could be a configurable parameter)
• selectColumnsToIndex: this method is called every time we build a new table, only once (if we do not want to update it).

Checklist:

Here is the list of things you should do before submitting this pull request:

• Add Skeleton for extensible AutoIndexing code (support for different algorithms and engines)
• Add default PCA code
• Add tests.
• Add comments to the code (make it easier for the community!).
• Change the documentation.
• Your branch is updated to the main-1.0.0 branch (dependent changes have been merged).

How Has This Been Tested? (Optional)

This should be tested individually (each AutoIndexer should have unit tests) as well as Integrated with the Spark API (DataFrame reads and writes).

[codecov]

Codecov Report

Attention: 17 lines in your changes are missing coverage. Please review.

Comparison is base (18b4534) 91.76% compared to head (e423a66) 91.02%.
Report is 52 commits behind head on main-1.0.0.

Files	Patch %	Lines
.../main/scala/io/qbeast/spark/delta/IndexFiles.scala	94.20%	4 Missing ⚠️
...cala/io/qbeast/spark/index/OTreeDataAnalyzer.scala	94.73%	3 Missing ⚠️
...cala/io/qbeast/core/model/CubeDomainsBuilder.scala	97.53%	2 Missing ⚠️
...o/qbeast/spark/delta/writer/RollupDataWriter.scala	98.44%	2 Missing ⚠️
...re/src/main/scala/io/qbeast/core/model/Block.scala	94.44%	1 Missing ⚠️
src/main/scala/io/qbeast/spark/QbeastTable.scala	50.00%	1 Missing ⚠️
...ala/io/qbeast/spark/delta/writer/BlockWriter.scala	96.96%	1 Missing ⚠️
...in/scala/io/qbeast/spark/delta/writer/Rollup.scala	96.00%	1 Missing ⚠️
...la/io/qbeast/spark/index/query/QueryExecutor.scala	90.90%	1 Missing ⚠️
src/main/scala/io/qbeast/spark/utils/Params.scala	50.00%	1 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff               @@
##           main-1.0.0     #247      +/-   ##
==============================================
- Coverage       91.76%   91.02%   -0.75%     
==============================================
  Files              91       95       +4     
  Lines            2258     2528     +270     
  Branches          167      323     +156     
==============================================
+ Hits             2072     2301     +229     
- Misses            186      227      +41     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[osopardo1]

Codecov is failing, but test pass and PR is ready to be reviewed.