Skip to content

Commit

Permalink
docs: update lakefs spark client to v0.11.0 (#6865)
Browse files Browse the repository at this point in the history
  • Loading branch information
nopcoder authored Oct 25, 2023
1 parent 75a2035 commit f86b7fa
Show file tree
Hide file tree
Showing 3 changed files with 8 additions and 8 deletions.
2 changes: 1 addition & 1 deletion docs/howto/export.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ The complete `spark-submit` command would look as follows:
spark-submit --conf spark.hadoop.lakefs.api.url=https://<LAKEFS_ENDPOINT>/api/v1 \
--conf spark.hadoop.lakefs.api.access_key=<LAKEFS_ACCESS_KEY_ID> \
--conf spark.hadoop.lakefs.api.secret_key=<LAKEFS_SECRET_ACCESS_KEY> \
--packages io.lakefs:lakefs-spark-client_2.12:0.10.0 \
--packages io.lakefs:lakefs-spark-client_2.12:0.11.0 \
--class io.treeverse.clients.Main export-app example-repo s3://example-bucket/prefix \
--branch=example-branch
```
Expand Down
8 changes: 4 additions & 4 deletions docs/howto/garbage-collection/gc.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,7 @@ spark-submit --class io.treeverse.gc.GarbageCollection \
-c spark.hadoop.lakefs.api.secret_key=<LAKEFS_SECRET_KEY> \
-c spark.hadoop.fs.s3a.access.key=<S3_ACCESS_KEY> \
-c spark.hadoop.fs.s3a.secret.key=<S3_SECRET_KEY> \
http://treeverse-clients-us-east.s3-website-us-east-1.amazonaws.com/lakefs-spark-client/0.10.0/lakefs-spark-client-assembly-0.10.0.jar \
http://treeverse-clients-us-east.s3-website-us-east-1.amazonaws.com/lakefs-spark-client/0.11.0/lakefs-spark-client-assembly-0.11.0.jar \
example-repo us-east-1
```
</div>
Expand All @@ -136,7 +136,7 @@ spark-submit --class io.treeverse.gc.GarbageCollection \
-c spark.hadoop.lakefs.api.access_key=<LAKEFS_ACCESS_KEY> \
-c spark.hadoop.lakefs.api.secret_key=<LAKEFS_SECRET_KEY> \
-c spark.hadoop.fs.azure.account.key.<AZURE_STORAGE_ACCOUNT>.dfs.core.windows.net=<AZURE_STORAGE_ACCESS_KEY> \
http://treeverse-clients-us-east.s3-website-us-east-1.amazonaws.com/lakefs-spark-client/0.10.0/lakefs-spark-client-assembly-0.10.0.jar \
http://treeverse-clients-us-east.s3-website-us-east-1.amazonaws.com/lakefs-spark-client/0.11.0/lakefs-spark-client-assembly-0.11.0.jar \
example-repo
```

Expand All @@ -153,7 +153,7 @@ spark-submit --class io.treeverse.gc.GarbageCollection \
-c spark.hadoop.fs.azure.account.oauth2.client.id.<AZURE_STORAGE_ACCOUNT>.dfs.core.windows.net=<application-id> \
-c spark.hadoop.fs.azure.account.oauth2.client.secret.<AZURE_STORAGE_ACCOUNT>.dfs.core.windows.net=<service-credential-key> \
-c spark.hadoop.fs.azure.account.oauth2.client.endpoint.<AZURE_STORAGE_ACCOUNT>.dfs.core.windows.net=https://login.microsoftonline.com/<directory-id>/oauth2/token \
http://treeverse-clients-us-east.s3-website-us-east-1.amazonaws.com/lakefs-spark-client/0.10.0/lakefs-spark-client-assembly-0.10.0.jar \
http://treeverse-clients-us-east.s3-website-us-east-1.amazonaws.com/lakefs-spark-client/0.11.0/lakefs-spark-client-assembly-0.11.0.jar \
example-repo
```

Expand Down Expand Up @@ -181,7 +181,7 @@ spark-submit --class io.treeverse.gc.GarbageCollection \
-c spark.hadoop.fs.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem \
-c spark.hadoop.fs.AbstractFileSystem.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS \
-c spark.hadoop.lakefs.gc.do_sweep=false \
http://treeverse-clients-us-east.s3-website-us-east-1.amazonaws.com/lakefs-spark-client/0.10.0/lakefs-spark-client-assembly-0.10.0.jar \
http://treeverse-clients-us-east.s3-website-us-east-1.amazonaws.com/lakefs-spark-client/0.11.0/lakefs-spark-client-assembly-0.11.0.jar \
example-repo
```

Expand Down
6 changes: 3 additions & 3 deletions docs/reference/spark-client.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,13 @@ Start Spark Shell / PySpark with the `--packages` flag:

This client is compiled for Spark 3.1.2 with Hadoop 3.2.1, but can work for other Spark
versions and higher Hadoop versions.

```bash
spark-shell --packages io.lakefs:lakefs-spark-client_2.12:0.10.0
spark-shell --packages io.lakefs:lakefs-spark-client_2.12:0.11.0
```

Alternatively an assembled jar is available on S3, at
`s3://treeverse-clients-us-east/lakefs-spark-client/0.10.0/lakefs-spark-client-assembly-0.10.0.jar`
`s3://treeverse-clients-us-east/lakefs-spark-client/0.11.0/lakefs-spark-client-assembly-0.11.0.jar`
</div>

## Configuration
Expand Down

0 comments on commit f86b7fa

Please sign in to comment.