Use env var in spark container (#1522)

* added

Signed-off-by: owenowenisme <mses010108@gmail.com>

* fix

Signed-off-by: owenowenisme <mses010108@gmail.com>

* add export

Signed-off-by: owenowenisme <mses010108@gmail.com>

* update docs using .env

Signed-off-by: owenowenisme <mses010108@gmail.com>

* update docs

Signed-off-by: owenowenisme <mses010108@gmail.com>

* change back from using .env to export

Signed-off-by: owenowenisme <mses010108@gmail.com>

* Apply suggestions from code review

Co-authored-by: Adnan Hemani <adnan.h@berkeley.edu>

---------

Signed-off-by: owenowenisme <mses010108@gmail.com>
Co-authored-by: Adnan Hemani <adnan.h@berkeley.edu>

Author: owenowenisme

getting-started/assets/polaris/create-catalog.sh

@@ -22,7 +22,7 @@ set -e
 apk add --no-cache jq
 
 token=$(curl -s http://polaris:8181/api/catalog/v1/oauth/tokens \
-  --user root:s3cr3t \
+  --user ${CLIENT_ID}:${CLIENT_SECRET} \
   -d grant_type=client_credentials \
   -d scope=PRINCIPAL_ROLE:ALL | sed -n 's/.*"access_token":"\([^"]*\)".*/\1/p')
 

getting-started/assets/trino-config/catalog/iceberg.properties

@@ -21,7 +21,7 @@ connector.name=iceberg
 iceberg.catalog.type=rest
 iceberg.rest-catalog.uri=http://polaris:8181/api/catalog
 iceberg.rest-catalog.security=OAUTH2
-iceberg.rest-catalog.oauth2.credential=root:s3cr3t
+iceberg.rest-catalog.oauth2.credential=${ENV:CLIENT_ID}:${ENV:CLIENT_SECRET}
 iceberg.rest-catalog.oauth2.scope=PRINCIPAL_ROLE:ALL
 iceberg.rest-catalog.warehouse=quickstart_catalog
 # Required to support local filesystem: https://trino.io/docs/current/object-storage.html#configuration

getting-started/eclipselink/docker-compose.yml

@@ -36,6 +36,7 @@ services:
       polaris.persistence.eclipselink.configuration-file: /deployments/config/eclipselink/persistence.xml
       polaris.realm-context.realms: POLARIS
       quarkus.otel.sdk.disabled: "true"
+      POLARIS_BOOTSTRAP_CREDENTIALS: POLARIS,${CLIENT_ID},${CLIENT_SECRET}
     volumes:
       - ../assets/eclipselink/:/deployments/config/eclipselink
     healthcheck:
@@ -54,6 +55,8 @@ services:
       - STORAGE_LOCATION=${STORAGE_LOCATION}
       - AWS_ROLE_ARN=${AWS_ROLE_ARN}
       - AZURE_TENANT_ID=${AZURE_TENANT_ID}
+      - CLIENT_ID=${CLIENT_ID}
+      - CLIENT_SECRET=${CLIENT_SECRET}
     volumes:
       - ../assets/polaris/:/polaris
     entrypoint: '/bin/sh -c "chmod +x /polaris/create-catalog.sh && /polaris/create-catalog.sh"'
@@ -79,7 +82,7 @@ services:
       --conf, "spark.sql.catalog.quickstart_catalog.type=rest",
       --conf, "spark.sql.catalog.quickstart_catalog.warehouse=quickstart_catalog",
       --conf, "spark.sql.catalog.quickstart_catalog.uri=http://polaris:8181/api/catalog",
-      --conf, "spark.sql.catalog.quickstart_catalog.credential=root:s3cr3t",
+      --conf, "spark.sql.catalog.quickstart_catalog.credential=${USER_CLIENT_ID}:${USER_CLIENT_SECRET}",
       --conf, "spark.sql.catalog.quickstart_catalog.scope=PRINCIPAL_ROLE:ALL",
       --conf, "spark.sql.defaultCatalog=quickstart_catalog",
       --conf, "spark.sql.catalogImplementation=in-memory",
@@ -91,6 +94,9 @@ services:
     depends_on:
       polaris-setup:
         condition: service_completed_successfully
+    environment:
+      - CLIENT_ID=${CLIENT_ID}
+      - CLIENT_SECRET=${CLIENT_SECRET}
     stdin_open: true
     tty: true
     ports:

site/content/in-dev/unreleased/getting-started/using-polaris.md

@@ -21,7 +21,12 @@ Title: Using Polaris
 type: docs
 weight: 400
 ---
-
+## Setup
+Define your `CLIENT_ID` & `CLIENT_SECRET` and export them for future use.
+```shell
+export CLIENT_ID=YOUR_CLIENT_ID
+export CLIENT_SECRET=YOUR_CLIENT_SECRET
+```
 ## Defining a Catalog
 
 In Polaris, the [catalog]({{% relref "../entities#catalog" %}}) is the top-level entity that objects like [tables]({{% relref "../entities#table" %}}) and [views]({{% relref "../entities#view" %}}) are organized under. With a Polaris service running, you can create a catalog like so:
@@ -79,11 +84,13 @@ With a catalog created, we can create a [principal]({{% relref "../entities#prin
 
 Be sure to provide the necessary credentials, hostname, and port as before.
 
-When the `principals create` command completes successfully, it will return the credentials for this new principal. Be sure to note these down for later. For example:
+When the `principals create` command completes successfully, it will return the credentials for this new principal. Export them for future use. For example:
 
-```
+```shell
 ./polaris ... principals create example
 {"clientId": "XXXX", "clientSecret": "YYYY"}
+export USER_CLIENT_ID=XXXX
+export USER_CLIENT_SECRET=YYYY
 ```
 
 Now, we grant the principal the [principal role]({{% relref "../entities#principal-role" %}}) we created, and grant the [catalog role]({{% relref "../entities#catalog-role" %}}) the principal role we created. For more information on these entities, please refer to the linked documentation.
@@ -154,27 +161,21 @@ bin/spark-sql \
 --conf spark.sql.catalog.quickstart_catalog=org.apache.iceberg.spark.SparkCatalog \
 --conf spark.sql.catalog.quickstart_catalog.catalog-impl=org.apache.iceberg.rest.RESTCatalog \
 --conf spark.sql.catalog.quickstart_catalog.uri=http://localhost:8181/api/catalog \
---conf spark.sql.catalog.quickstart_catalog.credential='XXXX:YYYY' \
+--conf spark.sql.catalog.quickstart_catalog.credential='${USER_CLIENT_ID}:${USER_CLIENT_SECRET}' \
 --conf spark.sql.catalog.quickstart_catalog.scope='PRINCIPAL_ROLE:ALL' \
 --conf spark.sql.catalog.quickstart_catalog.token-refresh-enabled=true \
 --conf spark.sql.catalog.quickstart_catalog.client.region=us-west-2
 ```
 
 
-Replace `XXXX` and `YYYY` with the client ID and client secret generated when you created the `quickstart_user` principal.
-
 Similar to the CLI commands above, this configures Spark to use the Polaris running at `localhost:8181`. If your Polaris server is running elsewhere, but sure to update the configuration appropriately.
 
 Finally, note that we include the `hadoop-aws` package here. If your table is using a different filesystem, be sure to include the appropriate dependency.
 
 #### Using Spark SQL from a Docker container
 
-Replace the credentials used in the Docker container using the following code:
-
+Refresh the Docker container with the user's credentials:
 ```shell
-USER_CLIENT_ID="XXXX"
-USER_CLIENT_SECRET="YYYY"
-sed -i "s/^\(.*spark\.sql\.catalog\.quickstart_catalog\.credential=\).*/\1${USER_CLIENT_ID}:${USER_CLIENT_SECRET}\",/" getting-started/eclipselink/docker-compose.yml
 docker compose -f getting-started/eclipselink/docker-compose.yml up -d
 ```
 
@@ -233,12 +234,9 @@ org.apache.iceberg.exceptions.ForbiddenException: Forbidden: Principal 'quicksta
 
 ### Connecting with Trino
 
-Replace the credentials used in the Docker container using the following code:
+Refresh the Docker container with the user's credentials:
 
 ```shell
-USER_CLIENT_ID="XXXX"
-USER_CLIENT_SECRET="YYYY"
-sed -i "s/^\(iceberg\.rest-catalog\.oauth2\.credential=\).*/\1${USER_CLIENT_ID}:${USER_CLIENT_SECRET}/" getting-started/eclipselink/trino-config/catalog/iceberg.properties
 docker compose -f getting-started/eclipselink/docker-compose.yml down trino
 docker compose -f getting-started/eclipselink/docker-compose.yml up -d
 ```