Unified Catalog

[osopardo1]

Description

In this PR, we are modifying the QbeastCatalog to process also writes in Delta (or other formats such Iceberg or Hudi once they're integrated).

Type of change

It is an improvement of an existing Catalog feature. Instead of managing two catalogs at the same time, the user would only need to set up one by adding the following configuration to any Spark Application:

--conf spark.sql.catalog.spark_catalog=io.qbeast.spark.internal.sources.catalog.QbeastCatalog

Then, the user can create qbeast tables and delta tables with either DataFrame API or SQL:

data.write.format("delta").saveAsTable("delta_table")

data.write
 .format("qbeast")
 .option("columnsToIndex", "id")
 .saveAsTable("qbeast_table")

spark.sql(
  s"CREATE TABLE student (id INT, name STRING, age INT) USING qbeast " +
    "OPTIONS ('columnsToIndex'='id')")

spark.sql(
  s"CREATE TABLE student (id INT, name STRING, age INT) USING delta")

The user can also choose to configure them as separate catalogs. But for some integrations (such as dbt) it's not recommended.

Checklist:

Here is the list of things you should do before submitting this pull request:

• New feature / bug fix has been committed following the Contribution guide.
• Add comments to the code (make it easier for the community!).
• Change the documentation.
• Add tests.
• Your branch is updated to the main branch (dependent changes have been merged).

How Has This Been Tested? (Optional)

Please describe the tests that you ran to verify your changes.

We add a test in QbeastCatalogIntegrationTest in which we try to write as delta and qbeast with the same Catalog.

[osopardo1]

Hello! I would suggest @Adricu8 and @eavilaes try the solution in your DBT project.
You can do an sbt assembly on the code and test it. This assembly would contain both qbeast-spark and qbeast-core packages. I will also put you as reviewers to verify everything works.

UPDATE: You can also retrieve the package of the new qbeast-spark SNAPSHOT from https://s01.oss.sonatype.org/content/repositories/snapshots/io/qbeast/qbeast-spark_2.12/0.3.0-alpha-SNAPSHOT/qbeast-spark_2.12-0.3.0-alpha-SNAPSHOT.jar
And qbeast-core https://s01.oss.sonatype.org/content/repositories/snapshots/io/qbeast/qbeast-spark_2.12/0.3.0-alpha-SNAPSHOT/qbeast-core_2.12-0.3.0-alpha-SNAPSHOT.jar

Thank you!

[codecov]

Codecov Report

Merging #143 (83e3e9a) into main (d503833) will decrease coverage by 0.02%.
The diff coverage is 100.00%.

❗ Current head 83e3e9a differs from pull request most recent head c0e9f5f. Consider uploading reports for the commit c0e9f5f to get more accurate results

@@            Coverage Diff             @@
##             main     #143      +/-   ##
==========================================
- Coverage   92.86%   92.83%   -0.03%     
==========================================
  Files          73       73              
  Lines        1709     1717       +8     
  Branches      126      126              
==========================================
+ Hits         1587     1594       +7     
- Misses        122      123       +1     

Impacted Files	Coverage Δ
...spark/internal/sources/catalog/QbeastCatalog.scala	100.00% <100.00%> (ø)
...ala/io/qbeast/spark/delta/writer/BlockWriter.scala	95.65% <0.00%> (-2.18%)	⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[osopardo1]

In this PR, I also updated the docs and I will change the version to 0.3.0.
Ideally, we can publish the new version before Christmas 😃