Skip to content

Commit d035344

Browse files
Spark 3.5.6 and Iceberg 1.9.1 (#1960)
* Spark 3.5.6 and Iceberg 1.9.1 * Cleanup
1 parent 05975a3 commit d035344

File tree

21 files changed

+55
-55
lines changed

21 files changed

+55
-55
lines changed

.github/workflows/spark_client_regtests.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@ jobs:
6464
:polaris-server:quarkusAppPartsBuild --rerun \
6565
-Dquarkus.container-image.build=true
6666
67-
# NOTE: the regression test runs with spark 3.5.5 and scala 2.12 in Java 17. We also have integration
67+
# NOTE: the regression test runs with spark 3.5.6 and scala 2.12 in Java 17. We also have integration
6868
# tests runs with the existing gradle.yml, which only runs on Java 21. Since spark Java compatibility
6969
# for 3.5 is 8, 11, and 17, we should run spark client with those compatible java versions.
7070
# TODO: add separate spark client CI and run with Java 8, 11 and 17.

getting-started/eclipselink/docker-compose.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -79,7 +79,7 @@ services:
7979
retries: 15
8080
command: [
8181
/opt/spark/bin/spark-sql,
82-
--packages, "org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.9.0,org.apache.iceberg:iceberg-aws-bundle:1.9.0,org.apache.iceberg:iceberg-gcp-bundle:1.9.0,org.apache.iceberg:iceberg-azure-bundle:1.9.0",
82+
--packages, "org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.9.1,org.apache.iceberg:iceberg-aws-bundle:1.9.1,org.apache.iceberg:iceberg-gcp-bundle:1.9.1,org.apache.iceberg:iceberg-azure-bundle:1.9.1",
8383
--conf, "spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions",
8484
--conf, "spark.sql.catalog.quickstart_catalog=org.apache.iceberg.spark.SparkCatalog",
8585
--conf, "spark.sql.catalog.quickstart_catalog.type=rest",

getting-started/jdbc/docker-compose.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,7 @@ services:
8181
retries: 15
8282
command: [
8383
/opt/spark/bin/spark-sql,
84-
--packages, "org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.9.0,org.apache.iceberg:iceberg-aws-bundle:1.9.0,org.apache.iceberg:iceberg-gcp-bundle:1.9.0,org.apache.iceberg:iceberg-azure-bundle:1.9.0",
84+
--packages, "org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.9.1,org.apache.iceberg:iceberg-aws-bundle:1.9.1,org.apache.iceberg:iceberg-gcp-bundle:1.9.1,org.apache.iceberg:iceberg-azure-bundle:1.9.1",
8585
--conf, "spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions",
8686
--conf, "spark.sql.catalog.polaris=org.apache.iceberg.spark.SparkCatalog",
8787
--conf, "spark.sql.catalog.polaris.type=rest",

getting-started/spark/notebooks/SparkPolaris.ipynb

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -256,7 +256,7 @@
256256
"\n",
257257
"spark = (SparkSession.builder\n",
258258
" .config(\"spark.sql.catalog.spark_catalog\", \"org.apache.iceberg.spark.SparkSessionCatalog\")\n",
259-
" .config(\"spark.jars.packages\", \"org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.9.0,org.apache.iceberg:iceberg-aws-bundle:1.9.0\")\n",
259+
" .config(\"spark.jars.packages\", \"org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.9.1,org.apache.iceberg:iceberg-aws-bundle:1.9.1\")\n",
260260
" .config('spark.sql.iceberg.vectorization.enabled', 'false')\n",
261261
" \n",
262262
" # Configure the 'polaris' catalog as an Iceberg rest catalog\n",

gradle/libs.versions.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@
2020
[versions]
2121
checkstyle = "10.25.0"
2222
hadoop = "3.4.1"
23-
iceberg = "1.9.0" # Ensure to update the iceberg version in regtests to keep regtests up-to-date
23+
iceberg = "1.9.1" # Ensure to update the iceberg version in regtests to keep regtests up-to-date
2424
quarkus = "3.24.3"
2525
immutables = "2.11.1"
2626
picocli = "4.7.7"

plugins/pluginlibs.versions.toml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@
1818
#
1919

2020
[versions]
21-
iceberg = "1.9.0"
22-
spark35 = "3.5.5"
21+
iceberg = "1.9.1"
22+
spark35 = "3.5.6"
2323
scala212 = "2.12.19"
2424
scala213 = "2.13.15"

plugins/spark/README.md

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -21,12 +21,12 @@
2121

2222
The Polaris Spark plugin provides a SparkCatalog class, which communicates with the Polaris
2323
REST endpoints, and provides implementations for Apache Spark's
24-
[TableCatalog](https://github.com/apache/spark/blob/v3.5.5/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCatalog.java),
25-
[SupportsNamespaces](https://github.com/apache/spark/blob/v3.5.5/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/SupportsNamespaces.java),
26-
[ViewCatalog](https://github.com/apache/spark/blob/v3.5.5/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/ViewCatalog.java) classes.
24+
[TableCatalog](https://github.com/apache/spark/blob/v3.5.6/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCatalog.java),
25+
[ViewCatalog](https://github.com/apache/spark/blob/v3.5.6/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/ViewCatalog.java) classes.
26+
[SupportsNamespaces](https://github.com/apache/spark/blob/v3.5.6/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/SupportsNamespaces.java),
2727

2828
Right now, the plugin only provides support for Spark 3.5, Scala version 2.12 and 2.13,
29-
and depends on iceberg-spark-runtime 1.9.0.
29+
and depends on iceberg-spark-runtime 1.9.1.
3030

3131
# Start Spark with local Polaris service using the Polaris Spark plugin
3232
The following command starts a Polaris server for local testing, it runs on localhost:8181 with default
@@ -50,7 +50,7 @@ Run the following command to build the Polaris Spark project and publish the sou
5050

5151
```shell
5252
bin/spark-shell \
53-
--packages org.apache.polaris:polaris-spark-<spark_version>_<scala_version>:<polaris_version>,org.apache.iceberg:iceberg-aws-bundle:1.9.0,io.delta:delta-spark_2.12:3.3.1 \
53+
--packages org.apache.polaris:polaris-spark-<spark_version>_<scala_version>:<polaris_version>,org.apache.iceberg:iceberg-aws-bundle:1.9.1,io.delta:delta-spark_2.12:3.3.1 \
5454
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions,io.delta.sql.DeltaSparkSessionExtension \
5555
--conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog \
5656
--conf spark.sql.catalog.<catalog-name>.warehouse=<catalog-name> \
@@ -73,7 +73,7 @@ The Spark command would look like following:
7373

7474
```shell
7575
bin/spark-shell \
76-
--packages org.apache.polaris:polaris-spark-3.5_2.12:1.1.0-incubating-SNAPSHOT,org.apache.iceberg:iceberg-aws-bundle:1.9.0,io.delta:delta-spark_2.12:3.3.1 \
76+
--packages org.apache.polaris:polaris-spark-3.5_2.12:1.1.0-incubating-SNAPSHOT,org.apache.iceberg:iceberg-aws-bundle:1.9.1,io.delta:delta-spark_2.12:3.3.1 \
7777
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions,io.delta.sql.DeltaSparkSessionExtension \
7878
--conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog \
7979
--conf spark.sql.catalog.polaris.warehouse=polaris \
@@ -99,7 +99,7 @@ To start Spark using the bundle JAR, specify it with the `--jars` option as show
9999
```shell
100100
bin/spark-shell \
101101
--jars <path-to-spark-client-jar> \
102-
--packages org.apache.iceberg:iceberg-aws-bundle:1.9.0,io.delta:delta-spark_2.12:3.3.1 \
102+
--packages org.apache.iceberg:iceberg-aws-bundle:1.9.1,io.delta:delta-spark_2.12:3.3.1 \
103103
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions,io.delta.sql.DeltaSparkSessionExtension \
104104
--conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog \
105105
--conf spark.sql.catalog.<catalog-name>.warehouse=<catalog-name> \

plugins/spark/v3.5/getting-started/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,7 @@ This will spin up 2 container services
5252
* The `polaris` service for running Apache Polaris using an in-memory metastore
5353
* The `jupyter` service for running Jupyter notebook with PySpark
5454

55-
NOTE: Starting the container first time may take a couple of minutes, because it will need to download the Spark 3.5.5.
55+
NOTE: Starting the container first time may take a couple of minutes, because it will need to download the Spark 3.5.6.
5656
When working with Delta, the Polaris Spark Client requires delta-io >= 3.2.1, and it requires at least Spark 3.5.3,
5757
but the current jupyter Spark image only support Spark 3.5.0.
5858

plugins/spark/v3.5/getting-started/notebooks/Dockerfile

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -24,11 +24,11 @@ ENV LANGUAGE='en_US:en'
2424
USER root
2525

2626
# Generic table support requires delta 3.2.1
27-
# Install Spark 3.5.5
28-
RUN wget -q https://www.apache.org/dyn/closer.lua/spark/spark-3.5.5/spark-3.5.5-bin-hadoop3.tgz?action=download \
29-
&& tar -xzf spark-3.5.5-bin-hadoop3.tgz \
30-
&& mv spark-3.5.5-bin-hadoop3 /opt/spark \
31-
&& rm spark-3.5.5-bin-hadoop3.tgz
27+
# Install Spark 3.5.6
28+
RUN wget -q https://www.apache.org/dyn/closer.lua/spark/spark-3.5.6/spark-3.5.6-bin-hadoop3.tgz?action=download \
29+
&& tar -xzf spark-3.5.6-bin-hadoop3.tgz \
30+
&& mv spark-3.5.6-bin-hadoop3 /opt/spark \
31+
&& rm spark-3.5.6-bin-hadoop3.tgz
3232

3333
# Set environment variables
3434
ENV SPARK_HOME=/opt/spark

plugins/spark/v3.5/getting-started/notebooks/SparkPolaris.ipynb

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -266,7 +266,7 @@
266266
"\n",
267267
"spark = (SparkSession.builder\n",
268268
" .config(\"spark.jars\", \"../polaris_libs/polaris-spark-3.5_2.12-1.1.0-incubating-SNAPSHOT-bundle.jar\") # TODO: add a way to automatically discover the Jar\n",
269-
" .config(\"spark.jars.packages\", \"org.apache.iceberg:iceberg-aws-bundle:1.9.0,io.delta:delta-spark_2.12:3.2.1\")\n",
269+
" .config(\"spark.jars.packages\", \"org.apache.iceberg:iceberg-aws-bundle:1.9.1,io.delta:delta-spark_2.12:3.2.1\")\n",
270270
" .config(\"spark.sql.catalog.spark_catalog\", \"org.apache.spark.sql.delta.catalog.DeltaCatalog\")\n",
271271
" .config('spark.sql.iceberg.vectorization.enabled', 'false')\n",
272272
"\n",

0 commit comments

Comments
 (0)