Docs/improve idp documentation (#2695)

* Fix Github links in IDP documentation

* Separate IDP docs for usage and development

* - Add telemetry config example
- Fix link to getting started from landing page
- Fix mentioning role-arn as required

* Fix some relative links (local Hugo resolves them properly, but PR auto checks still fails)

* Docs: narrow down --role-arn usage for AWS S3 only; fix a link in keycloak guide.

* Docs: fix a link in keycloak guide.

Author: olsoloviov

client/python/cli/command/__init__.py

@@ -188,7 +188,5 @@ def validate(self) -> None:
         """
         Used to validate a command. Should always be called before `execute`. The arg parser will catch many issues
         with options, but this is used to apply additional constraints that the arg parser can't currently handle.
-        One example is that a catalog cannot be created with the `s3` storage type without a `--role-arn` option, but
-        one can be created without this flag if it's using the `gcs` storage type.
         """
         raise Exception("`validate` called on abstract `Command`")

client/python/cli/constants.py

@@ -227,7 +227,7 @@ class Create:
                 "Multiple locations can be provided by specifying this option more than once."
             )
 
-            ROLE_ARN = "(Only for S3) A role ARN to use when connecting to S3"
+            ROLE_ARN = "(Only for AWS S3) A role ARN to use when connecting to S3"
             EXTERNAL_ID = "(Only for S3) The external ID to use when connecting to S3"
             REGION = "(Only for S3) The region to use when connecting to S3"
             USER_ARN = "(Only for S3) A user ARN to use when connecting to S3"

site/content/_index.adoc

@@ -25,7 +25,7 @@ cascade:
 {{< blocks/cover title="Welcome to the Apache Polaris™ (incubating) web site!" image_anchor="center" color="primary" >}}
 Apache Polaris is an open-source, fully-featured catalog for Apache Iceberg™. It implements Iceberg's REST API, enabling seamless multi-engine interoperability across a wide range of platforms, including Apache Doris™, Apache Flink®, Apache Spark™, Dremio® OSS, StarRocks, and Trino.
 
-<a href="/in-dev/unreleased/getting-started/install-dependencies/" class="btn btn-lg btn-dark mt-5">Get Started <i class="fas fa-arrow-alt-circle-right ms-2"></i></a>
+<a href="/in-dev/unreleased/getting-started/" class="btn btn-lg btn-dark mt-5">Get Started <i class="fas fa-arrow-alt-circle-right ms-2"></i></a>
 
 {{< /blocks/cover >}}
 

site/content/in-dev/unreleased/command-line-interface.md

@@ -133,8 +133,12 @@ options:
       --type  The type of catalog to create in [INTERNAL, EXTERNAL]. INTERNAL by default.
       --storage-type  (Required) The type of storage to use for the catalog
       --default-base-location  (Required) Default base location of the catalog
+      --endpoint  (Only for S3) The S3 endpoint to use when connecting to S3
+      --endpoint-internal  (Only for S3) The S3 endpoint used by Polaris to use when connecting to S3, if different from the one that clients use
+      --sts-endpoint  (Only for S3) The STS endpoint to use when connecting to STS
+      --path-style-access  (Only for S3) Whether to use path-style-access for S3
       --allowed-location  An allowed location for files tracked by the catalog. Multiple locations can be provided by specifying this option more than once.
-      --role-arn  (Required for S3) A role ARN to use when connecting to S3
+      --role-arn  (Only for AWS S3) A role ARN to use when connecting to S3
       --region  (Only for S3) The region to use when connecting to S3
       --external-id  (Only for S3) The external ID to use when connecting to S3
       --tenant-id  (Required for Azure) A tenant ID to use when connecting to Azure Storage
@@ -145,7 +149,7 @@ options:
       --catalog-connection-type  The type of external catalog in [ICEBERG, HADOOP].
       --iceberg-remote-catalog-name  The remote catalog name when federating to an Iceberg REST catalog
       --hadoop-warehouse  The warehouse to use when federating to a HADOOP catalog
-      --catalog-authentication-type  The type of authentication in [OAUTH, BEARER, SIGV4]
+      --catalog-authentication-type  The type of authentication in [OAUTH, BEARER, SIGV4, IMPLICIT]
       --catalog-service-identity-type  The type of service identity in [AWS_IAM]
       --catalog-service-identity-iam-arn  When using the AWS_IAM service identity type, this is the ARN of the IAM user or IAM role Polaris uses to assume roles and then access external resources.
       --catalog-uri  The URI of the external catalog

site/content/in-dev/unreleased/getting-started/creating-a-catalog/s3/_index.md

@@ -32,7 +32,7 @@ For the `polaris catalogs create` [command]({{% ref "../../../command-line-inter
 
 ```text
 --storage-type s3
---role-arn  (Required for S3) A role ARN to use when connecting to S3
+--role-arn  (Only for AWS S3) A role ARN to use when connecting to S3
 --region  (Only for S3) The region to use when connecting to S3
 --external-id  (Only for S3) The external ID to use when connecting to S3
 ```

site/content/in-dev/unreleased/getting-started/using-polaris.md renamed to site/content/in-dev/unreleased/getting-started/using-polaris/_index.md

@@ -19,7 +19,7 @@
 #
 Title: Using Polaris
 type: docs
-weight: 400
+weight: 401
 ---
 
 ## Setup
@@ -34,17 +34,17 @@ export CLIENT_SECRET=YOUR_CLIENT_SECRET
 Refer to the [Creating a Catalog]({{% ref "creating-a-catalog" %}}) page for instructions on defining a
 catalog for your specific storage type. The following examples assume the catalog's name is `quickstart_catalog`.
 
-In Polaris, the [catalog]({{% relref "../entities#catalog" %}}) is the top-level entity that objects like [tables]({{% relref "../entities#table" %}}) and [views]({{% relref "../entities#view" %}}) are organized under.
+In Polaris, the [catalog]({{% relref "../../entities#catalog" %}}) is the top-level entity that objects like [tables]({{% relref "../../entities#table" %}}) and [views]({{% relref "../../entities#view" %}}) are organized under.
 
 The `DEFAULT_BASE_LOCATION` value you provided at catalog creation time will be the default location that objects in
 this catalog should be stored in.
 
-Additionally, if Polaris is running somewhere other than `localhost:8181`, you can specify the correct hostname and port by providing `--host` and `--port` flags. For the full set of options supported by the CLI, please refer to the [docs]({{% relref "../command-line-interface" %}}).
+Additionally, if Polaris is running somewhere other than `localhost:8181`, you can specify the correct hostname and port by providing `--host` and `--port` flags. For the full set of options supported by the CLI, please refer to the [docs]({{% relref "../../command-line-interface" %}}).
 
 
 ### Creating a Principal and Assigning it Privileges
 
-With a catalog created, we can create a [principal]({{% relref "../entities#principal" %}}) that has access to manage that catalog. For details on how to configure the Polaris CLI, see [the section above](#defining-a-catalog) or refer to the [docs]({{% relref "../command-line-interface" %}}).
+With a catalog created, we can create a [principal]({{% relref "../../entities#principal" %}}) that has access to manage that catalog. For details on how to configure the Polaris CLI, see [the section above](#defining-a-catalog) or refer to the [docs]({{% relref "../../command-line-interface" %}}).
 
 ```shell
 ./polaris \
@@ -81,7 +81,7 @@ export USER_CLIENT_ID=XXXX
 export USER_CLIENT_SECRET=YYYY
 ```
 
-Now, we grant the principal the [principal role]({{% relref "../entities#principal-role" %}}) we created, and grant the [catalog role]({{% relref "../entities#catalog-role" %}}) the principal role we created. For more information on these entities, please refer to the linked documentation.
+Now, we grant the principal the [principal role]({{% relref "../../entities#principal-role" %}}) we created, and grant the [catalog role]({{% relref "../../entities#catalog-role" %}}) the principal role we created. For more information on these entities, please refer to the linked documentation.
 
 ```shell
 ./polaris \
@@ -106,7 +106,7 @@ Now, we’ve linked our principal to the catalog via roles like so:
 
 ![Principal to Catalog](/img/quickstart/privilege-illustration-1.png "Principal to Catalog")
 
-In order to give this principal the ability to interact with the catalog, we must assign some [privileges]({{% relref "../entities#privilege" %}}). For the time being, we will give this principal the ability to fully manage content in our new catalog. We can do this with the CLI like so:
+In order to give this principal the ability to interact with the catalog, we must assign some [privileges]({{% relref "../../entities#privilege" %}}). For the time being, we will give this principal the ability to fully manage content in our new catalog. We can do this with the CLI like so:
 
 ```shell
 ./polaris \
@@ -120,7 +120,7 @@ In order to give this principal the ability to interact with the catalog, we mus
   CATALOG_MANAGE_CONTENT
 ```
 
-This grants the [catalog privileges]({{% relref "../entities#privilege" %}}) `CATALOG_MANAGE_CONTENT` to our catalog role, linking everything together like so:
+This grants the [catalog privileges]({{% relref "../../entities#privilege" %}}) `CATALOG_MANAGE_CONTENT` to our catalog role, linking everything together like so:
 
 ![Principal to Catalog with Catalog Role](/img/quickstart/privilege-illustration-2.png "Principal to Catalog with Catalog Role")
 
@@ -334,7 +334,9 @@ curl -v http://127.0.0.1:8181/api/management/v1/catalogs/quickstart_catalog -H "
 ```
 
 ## Next Steps
-* Visit [Configuring Polaris for Production]({{% relref "../configuring-polaris-for-production" %}}).
+* Visit [Using Keycloak as the external identity provider]({{% relref "keycloak-idp" %}}).
+* Visit [Using Polaris with telemetry tools]({{% relref "telemetry-tools" %}}).
+* Visit [Configuring Polaris for Production]({{% relref "../../configuring-polaris-for-production" %}}).
 * A Getting Started experience for using Spark with Jupyter Notebooks is documented [here](https://github.com/apache/polaris/blob/main/getting-started/spark/README.md).
 * To shut down a locally-deployed Polaris server and clean up all related Docker containers, run the command listed below. Cloud Deployments have their respective termination commands on their Deployment page, while Polaris running on Gradle will terminate when the Gradle process terminates.
 ```shell

site/content/in-dev/unreleased/managing-security/external-idp/keycloak-idp.md renamed to site/content/in-dev/unreleased/getting-started/using-polaris/keycloak-idp.md

@@ -17,12 +17,14 @@
 # specific language governing permissions and limitations
 # under the License.
 #
-title: Using Polaris with Keycloak as external IDP
+title: Getting Started with Apache Polaris, External Authentication and Keycloak
 linkTitle: Using Keycloak IDP
 type: docs
-weight: 100
+weight: 400
 ---
 
+## Overview
+
 This example uses Keycloak as an **external** identity provider for Polaris. The "iceberg" realm is automatically
 created and configured from the `iceberg-realm.json` file.
 
@@ -48,7 +50,7 @@ Polaris is configured with 3 realms:
   issued by both Polaris and Keycloak.
 
 For more information about how to configure Polaris with external authentication, see the
-[Polaris documentation]({{% ref "../external-idp" %}}).
+[IDP integration documentation]({{% relref "../../managing-security/external-idp" %}}).
 
 ## Starting the Example
 

site/content/in-dev/unreleased/getting-started/using-polaris/telemetry-tools.md

@@ -0,0 +1,70 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Getting Started with Apache Polaris, Prometheus and Jaeger
+linkTitle: Using Polaris with telemetry tools
+type: docs
+weight: 401
+---
+
+This example requires `jq` to be installed on your machine.
+
+1. Build the Polaris image if it's not already present locally:
+
+    ```shell
+    ./gradlew \
+      :polaris-server:assemble \
+      :polaris-server:quarkusAppPartsBuild --rerun \
+      -Dquarkus.container-image.build=true
+    ```
+
+2. Start the docker compose group by running the following command from the root of the repository:
+
+    ```shell
+    export ASSETS_PATH=$(pwd)/getting-started/assets/
+    export CLIENT_ID=root
+    export CLIENT_SECRET=s3cr3t
+    docker compose -f getting-started/telemetry/docker-compose.yml up
+    ```
+
+3. To access Polaris from the host machine, first request an access token:
+
+    ```shell
+    export POLARIS_TOKEN=$(curl -s http://localhost:8181/api/catalog/v1/oauth/tokens \
+       --user root:s3cr3t \
+       -d 'grant_type=client_credentials' \
+       -d 'scope=PRINCIPAL_ROLE:ALL' | jq -r .access_token)
+    ```
+
+4. Then, use the access token in the Authorization header when accessing Polaris; you can also test
+   the `Polaris-Request-Id` header; you should see it in all logs and traces:
+
+    ```shell
+    curl -v 'http://localhost:8181/api/management/v1/principal-roles' \
+      -H "Authorization: Bearer $POLARIS_TOKEN" \
+      -H "Polaris-Request-Id: 1234"
+    curl -v 'http://localhost:8181/api/catalog/v1/config?warehouse=quickstart_catalog' \
+      -H "Authorization: Bearer $POLARIS_TOKEN" \
+      -H "Polaris-Request-Id: 5678"
+    ```
+
+5. Access the following services:
+
+    - Prometheus UI: browse to http://localhost:9093 to view metrics.
+    - Jaeger UI: browse to http://localhost:16686 to view traces.

site/content/in-dev/unreleased/managing-security/_index.md

@@ -20,5 +20,9 @@
 title: Managing Security
 linkTitle: Managing Security
 type: docs
-weight: 1001
+weight: 550
 ---
+
+## [Access Control]({{< relref "access-control" >}})
+
+## [Authentification and Identity Providers]({{< relref "external-idp" >}})