diff --git a/clients/spark/CHANGELOG.md b/clients/spark/CHANGELOG.md index b248af1d253..2a611aa1685 100644 --- a/clients/spark/CHANGELOG.md +++ b/clients/spark/CHANGELOG.md @@ -1,5 +1,8 @@ # Changelog +## v0.14.1 - 2024-07-04 +* When scanning across all commits, correctly handle lakeFS storage namespaces that do not end in `/` (#7955) + ## v0.14.0 - 2024-06-20 * Expose user metadata for commit (#7886) diff --git a/docs/howto/export.md b/docs/howto/export.md index 9e9b8a61576..3c3884c577c 100644 --- a/docs/howto/export.md +++ b/docs/howto/export.md @@ -55,7 +55,7 @@ The complete `spark-submit` command would look as follows: spark-submit --conf spark.hadoop.lakefs.api.url=https:///api/v1 \ --conf spark.hadoop.lakefs.api.access_key= \ --conf spark.hadoop.lakefs.api.secret_key= \ - --packages io.lakefs:lakefs-spark-client_2.12:0.14.0 \ + --packages io.lakefs:lakefs-spark-client_2.12:0.14.1 \ --class io.treeverse.clients.Main export-app example-repo s3://example-bucket/prefix \ --branch=example-branch ``` diff --git a/docs/howto/garbage-collection/gc.md b/docs/howto/garbage-collection/gc.md index ab9d96ef9b1..a57dc51434c 100644 --- a/docs/howto/garbage-collection/gc.md +++ b/docs/howto/garbage-collection/gc.md @@ -121,7 +121,7 @@ spark-submit --class io.treeverse.gc.GarbageCollection \ -c spark.hadoop.lakefs.api.secret_key= \ -c spark.hadoop.fs.s3a.access.key= \ -c spark.hadoop.fs.s3a.secret.key= \ - http://treeverse-clients-us-east.s3-website-us-east-1.amazonaws.com/lakefs-spark-client/0.14.0/lakefs-spark-client-assembly-0.14.0.jar \ + http://treeverse-clients-us-east.s3-website-us-east-1.amazonaws.com/lakefs-spark-client/0.14.1/lakefs-spark-client-assembly-0.14.1.jar \ example-repo us-east-1 ``` @@ -136,7 +136,7 @@ spark-submit --class io.treeverse.gc.GarbageCollection \ -c spark.hadoop.lakefs.api.access_key= \ -c spark.hadoop.lakefs.api.secret_key= \ -c spark.hadoop.fs.azure.account.key..dfs.core.windows.net= \ - http://treeverse-clients-us-east.s3-website-us-east-1.amazonaws.com/lakefs-spark-client/0.14.0/lakefs-spark-client-assembly-0.14.0.jar \ + http://treeverse-clients-us-east.s3-website-us-east-1.amazonaws.com/lakefs-spark-client/0.14.1/lakefs-spark-client-assembly-0.14.1.jar \ example-repo ``` @@ -153,7 +153,7 @@ spark-submit --class io.treeverse.gc.GarbageCollection \ -c spark.hadoop.fs.azure.account.oauth2.client.id..dfs.core.windows.net= \ -c spark.hadoop.fs.azure.account.oauth2.client.secret..dfs.core.windows.net= \ -c spark.hadoop.fs.azure.account.oauth2.client.endpoint..dfs.core.windows.net=https://login.microsoftonline.com//oauth2/token \ - http://treeverse-clients-us-east.s3-website-us-east-1.amazonaws.com/lakefs-spark-client/0.14.0/lakefs-spark-client-assembly-0.14.0.jar \ + http://treeverse-clients-us-east.s3-website-us-east-1.amazonaws.com/lakefs-spark-client/0.14.1/lakefs-spark-client-assembly-0.14.1.jar \ example-repo ``` @@ -181,7 +181,7 @@ spark-submit --class io.treeverse.gc.GarbageCollection \ -c spark.hadoop.fs.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem \ -c spark.hadoop.fs.AbstractFileSystem.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS \ -c spark.hadoop.lakefs.gc.do_sweep=false \ - http://treeverse-clients-us-east.s3-website-us-east-1.amazonaws.com/lakefs-spark-client/0.14.0/lakefs-spark-client-assembly-0.14.0.jar \ + http://treeverse-clients-us-east.s3-website-us-east-1.amazonaws.com/lakefs-spark-client/0.14.1/lakefs-spark-client-assembly-0.14.1.jar \ example-repo ``` diff --git a/docs/reference/spark-client.md b/docs/reference/spark-client.md index b1bc02bbf05..5c9a0d4d7f8 100644 --- a/docs/reference/spark-client.md +++ b/docs/reference/spark-client.md @@ -31,18 +31,18 @@ can work for other Spark versions and higher Hadoop versions. Start Spark Shell / PySpark with the `--packages` flag, for instance: ```bash -spark-shell --packages io.lakefs:lakefs-spark-client_2.12:0.14.0 +spark-shell --packages io.lakefs:lakefs-spark-client_2.12:0.14.1 ``` Alternatively use the assembled jar (an "Überjar") on S3, from -`s3://treeverse-clients-us-east/lakefs-spark-client/0.14.0/lakefs-spark-client-assembly-0.14.0.jar` +`s3://treeverse-clients-us-east/lakefs-spark-client/0.14.1/lakefs-spark-client-assembly-0.14.1.jar` by passing its path to `--jars`. The assembled jar is larger but shades several common libraries. Use it if Spark complains about bad classes or missing methods.
Include this assembled jar (an "Überjar") from S3, from -`s3://treeverse-clients-us-east/lakefs-spark-client/0.14.0/lakefs-spark-client-assembly-0.14.0.jar`. +`s3://treeverse-clients-us-east/lakefs-spark-client/0.14.1/lakefs-spark-client-assembly-0.14.1.jar`.