Standardize the way to ship new catalogs

[jackye1995]

This issue is opened to discuss about a standardized way to ship new catalogs, as we plan to ship GlueCatalog and NessieCatalog in upcoming release, and new catalogs like JDBC is also in progress to be added.

In the last community sync meeting, we discussed 2 ways:

1. for each new catalog (if in a new module), add a new runtime module that bundles all additional dependencies.
2. directly add the module in existing runtimes, such as iceberg-spark3-runtime as long as the jar size increase is reasonable.

For approach 1, compiling a runtime jar for shared usage seems to be not ideal. It will introduce duplicated class path issue on the user side, with the AWS client version used here potentially different from the one in the user's application. Another issue with this approach is that we will introduce a ton of runtime modules to iceberg as more catalogs are added, and this is not desired.

For approach 2, I did some experiments based on the current aws module, and the result was not good. When added to the spark3 runtime, the jar size increased from 18.9MB to 34.7MB, almost doubled. I checked all AWS dependencies, and even with all non-aws dependencies excluded, the added size was still over 10MB. I would imagine this situation very similar if we support for GCS and Azure are added in the future.

So the best way to go in open source seems to be not bundling any runtime jar, and a user can start the spark session by specifying the additional dependencies in the --packages flag:

spark-sql --packages org.apache.iceberg:iceberg-spark3-runtime:0.11.0,org.apache.iceberg:iceberg-aws:0.11.0,software.amazon.awssdk:bundle:2.15.40 \
    --conf spark.sql.catalog.test=org.apache.iceberg.spark.SparkCatalog \
    --conf spark.sql.catalog.test.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog \
    --conf spark.sql.catalog.test.warehouse=s3://some-bucket

By doing so, the user can freely choose the version of aws client, 2.15.40 here for example. And for users sensitive to jar size, they can cherry pick the aws client packages to bring in by themselves, instead of using the 250MB bundle.

Any thoughts? @rdblue @rymurr @yyanyy @giovannifumarola

[jacques-n]

Per my comments in the sync, I think a good standard approach would be:

• <=X, include directly in the base distribution jars
• >X, separate jar for this catalog modules

My vote would be to start with X == 1mb.

[rymurr]

My initial experiment with the Nessie catalog increased the runtime jar size by 500kb as almost all of its dependencies are already included in Spark. I think that is a reasonable increase (<3%) and I would agree w/ @jacques-n that a small increase in jar size is acceptable if it means we can ship more features to the user.

@jackye1995 is the Glue catalog small enough if included w/o dependencies? The user only having to add the aws bundle could be useful and we should be able to properly catch the resulting NoSuchMethodException and print a helpful error message if the user didn't add the bundle.

[jackye1995]

My vote would be to start with X == 1mb.

@jackye1995 is the Glue catalog small enough if included w/o dependencies?

Yeah that's actually a good way to go, basically mark all the AWS dependencies as provided. In that case the added jar size will only be the iceberg-aws itself which is 53KB.

So the user experience would be something like:

spark-sql --packages org.apache.iceberg:iceberg-spark3-runtime:0.11.0,software.amazon.awssdk:bundle:2.15.40 \
    --conf spark.sql.catalog.test=org.apache.iceberg.spark.SparkCatalog \
    --conf spark.sql.catalog.test.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog \
    --conf spark.sql.catalog.test.warehouse=s3://some-bucket

which looks good to me.

I will put up a PR for this after #1823 and #1844 are merged to change dependencies to compileOnly.

[rymurr]

I think it would be useful to try and catch any errors that may pop up if a user doesn't have the right classpath setup. I think clearly indicating that you have to add the aws bundle is a lot better than exploding (or worse silently swallowing). This is coming from someone who has spent entirely too much time trying to figure out why my spark job is failing ;-)

[rdblue]

I agree with what appears to be the consensus here, to add both Nessie and Glue to the existing runtime Jars and omit the AWS dependency because it is large.

I agree with @rymurr that if we have an opportunity to detect that AWS is missing and improve the error message, then that would be ideal.

Also, can we check to see if the AWS SDK is needed when running with the hadoop-aws module? That should provide the SDK and maybe we can just use that out of the box like we do with Hive.