[BUG] Getting the client-id and client-secret from stdout

[AlexMercedCoder]

A couple issues I had running through the quickstart, which just may documentation issues.

1. I deployed using Docker Compose

Looking at the stdout output I did not find what the docs asked me to look for:

**Bootstrapped with credentials: {"client-id": "XXXX", "client-secret": "YYYY"}**

But I did find this which I assumed was what I was looking for:

  realm: default-realm root principal credentials: fa44645a04410a0e:f1b82a42de2295da466682d3cfdbb0f1

I assume the format in this case was id:secret

So then ran the CLI command

./polaris \
  --client-id fa44645a04410a0e \
  --client-secret f1b82a42de2295da466682d3cfdbb0f1 \
  catalogs \
  create \
  --storage-type s3 \
  --default-base-location s3://some-example-bucket/folder/ \
  --role-arn arn:aws:iam::###########:role/polaris-storage \
  first-polaris-catalog

I ran into a few issues:

1. The Polaris CLI startup script did still needed Pydantic and dateutil to be installed, so I had to install those manually, assuming probably poetry. I do see them in the pyproject.toml, so I'm assuming that's more a "my system" problem, so Installed them in the virtual environment that was created and all seems to work fine now.

2. I ran the command above and got a JSON parsing error

Traceback (most recent call last):
  File "/home/alexmerced/development/developmentwork/dremio/repos/polaris/regtests/client/python/cli/polaris_cli.py", line 73, in <module>
    PolarisCli.execute()
  File "/home/alexmerced/development/developmentwork/dremio/repos/polaris/regtests/client/python/cli/polaris_cli.py", line 45, in execute
    error = json.loads(e.body)['error']
            ^^^^^^^^^^^^^^^^^^
  File "/home/alexmerced/.pyenv/versions/3.11.2/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alexmerced/.pyenv/versions/3.11.2/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alexmerced/.pyenv/versions/3.11.2/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

3. This wasn't super helpful so I modified polaris_cli.py to get me more detail

from cli.options.parser import Parser
from polaris.management import ApiClient, Configuration, ApiException
from polaris.management import PolarisDefaultApi

class PolarisCli:
    """
    Implements a basic Command-Line Interface (CLI) for interacting with a Polaris service. The CLI can be used to
    manage entities like catalogs, principals, and grants within Polaris and can perform most operations that are
    available in the Python client API.

    Example usage:
    * ./polaris --client-id ${id} --client-secret ${secret} --host ${hostname} principals create example_user
    * ./polaris --client-id ${id} --client-secret ${secret} --host ${hostname} principal-roles create example_role
    * ./polaris --client-id ${id} --client-secret ${secret} --host ${hostname} catalog-roles list
    """

    @staticmethod
    def execute():
        options = Parser.parse()
        client_builder = PolarisCli._get_client_builder(options)
        with client_builder() as api_client:
            try:
                from cli.command import Command
                admin_api = PolarisDefaultApi(api_client)
                command = Command.from_options(options)
                command.execute(admin_api)
            except ApiException as e:
                import json
                print(f'Exception when communicating with the Polaris server. Status code: {e.status}')
                print(f'Response headers: {e.headers}')
                print(f'Response body: {e.body}')
                try:
                    error = json.loads(e.body)['error']
                    print(f'{error["type"]}: {error["message"]}')
                except (json.JSONDecodeError, KeyError):
                    print('Error parsing response as JSON. Raw response body:')
                    print(e.body)

    @staticmethod
    def _get_client_builder(options):
        # Validate
        has_access_token = options.access_token is not None
        has_client_secret = options.client_id is not None and options.client_secret is not None
        if has_access_token and has_client_secret:
            raise Exception("Please provide credentials via either --client-id / --client-secret or "
                            "--access-token, but not both")

        # Authenticate accordingly
        polaris_catalog_url = f'http://{options.host}:{options.port}/api/management/v1'
        if has_access_token:
            return lambda: ApiClient(
                Configuration(host=polaris_catalog_url, access_token=options.access_token),
            )
        elif has_client_secret:
            return lambda: ApiClient(
                Configuration(host=polaris_catalog_url, username=options.client_id, password=options.client_secret),
            )
        else:
            raise Exception("Please provide credentials via --client-id & --client-secret or via --access-token")


if __name__ == '__main__':
    PolarisCli.execute()

3. So running this version of python_cli.py I got the following output:

Exception when communicating with the Polaris server. Status code: 401
Response headers: HTTPHeaderDict({'Date': 'Wed, 31 Jul 2024 15:16:14 GMT', 'Vary': 'Origin', 'WWW-Authenticate': 'Bearer realm="realm"', 'Content-Type': 'text/plain', 'Content-Length': '49'})
Response body: Credentials are required to access this resource.
Error parsing response as JSON. Raw response body:
Credentials are required to access this resource.

So I'm assuming the client-id and client-secret aren't what I thought they were or I passed them incorrectly, any guidance on this would be helpful.

[AlexMercedCoder]

Upon deeper exploration, based on the code I am correct that the values I thought were client and secret are, but when I started printing the headers/body/params of the request in the CLI it seemed that the client_id value and client_secret weren't being passed anywhere in the request. But when I printed (options.client_id) and (options.client_secret) in python_cli.py it did log the right values they just don't seem to be making it to the API call when I log the request details from rest.py .

Still working on it, if I can figure out the fix I'll make a PR.

[AlexMercedCoder]

Ok, so in rest.py I've logged

• params
• headers
• body
• the request object being sent here is what I get

##Params
{}
##Header
{'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/1.0.0/python'}
## Body
{"catalog": {"type": "INTERNAL", "name": "first-polaris-catalog", "properties": {"default-base-location": "s3://somebucket/folder/"}, "storageConfigInfo": {"storageType": "S3", "roleArn": "arn:aws:iam::xxxxxxxxxx:role/polaris-storage"}}}
##Request Object
r1 {'headers': HTTPHeaderDict({'Date': 'Wed, 31 Jul 2024 19:26:42 GMT', 'Vary': 'Origin', 'WWW-Authenticate': 'Bearer realm="realm"', 'Content-Type': 'text/plain', 'Content-Length': '49'}), 'status': 401, 'version': 11, 'reason': 'Unauthorized', 'strict': 0, 'decode_content': True, 'retries': Retry(total=3, connect=None, read=None, redirect=None, status=None), 'enforce_content_length': False, 'auto_close': True, '_decoder': None, '_body': None, '_fp': <http.client.HTTPResponse object at 0x75ec46e41840>, '_original_response': <http.client.HTTPResponse object at 0x75ec46e41840>, '_fp_bytes_read': 0, 'msg': None, '_request_url': 'http://localhost:8181/api/management/v1/catalogs', '_pool': <urllib3.connectionpool.HTTPConnectionPool object at 0x75ec46e785d0>, '_connection': <urllib3.connection.HTTPConnection object at 0x75ec47525750>, 'chunked': False, 'chunk_left': None, 'length_remaining': 49}

the r1 was print for me identify which request block in rest.py this was coming from which means it ran through this:

# no content type provided or payload is json
                content_type = headers.get('Content-Type')
                if (
                    not content_type
                    or re.search('json', content_type, re.IGNORECASE)
                ):
                    request_body = None
                    if body is not None:
                        request_body = json.dumps(body)
                        print(request_body)
                    r = self.pool_manager.request(
                        method,
                        url,
                        body=request_body,
                        timeout=timeout,
                        headers=headers,
                        preload_content=False
                    )
                    print("r1", r.__dict__)

Weird that the body was labeled as NONE cause when I logged request_body I got the right value and it's clearly assigned to body but when I log the request the body is not there.

[cgpoh]

Hi @AlexMercedCoder , I have the same error when I'm trying to create a catalog using CLI and following the readme doesn't help for me too. After digging further, I'm able to create my catalog using the following curl command:

curl -i -X POST -H "Authorization: Bearer $PRINCIPAL_TOKEN" -H 'Accept: application/json' -H 'Content-Type: application/json' \
  http://${POLARIS_HOST:-localhost}:8181/api/management/v1/catalogs \
  -d '{"name": "polaris", "type": "INTERNAL", "properties": {
        "default-base-location": "abfss://mycontainer@storageaccount.dfs.core.windows.net/test/"
    },"storageConfigInfo": {
        "tenantId": "long-tenant-id",
        "storageType": "AZURE",
        "allowedLocations": [
            "abfss://mycontainer@storageaccount.dfs.core.windows.net/test/"
        ]
    } }'

[AlexMercedCoder]

@cgpoh is the principal token just id:secret or do I have to make another curl request for the token cause I was working on trying to get the token endpoint to return me a token and hadn't successfully done that yet.

But this is super helpful, thank you!

[cgpoh]

@AlexMercedCoder, I have to make a curl request to get the token:

curl -i -X POST \
  http://localhost:8181/api/catalog/v1/oauth/tokens \
  -d 'grant_type=client_credentials&client_id=client_id_obtained_when_boot_up&client_secret=client_secret_obtained_when_boot_up&scope=PRINCIPAL_ROLE:ALL'

After getting the token:

{
    "access_token": "principal:root;password:af88b69c4f1215a1e62202ceb45f5a92;realm:default-realm;role:ALL",
    "scope": "PRINCIPAL_ROLE:ALL",
    "token_type": "bearer",
    "expires_in": 3600
}

I export the access_token:

export PRINCIPAL_TOKEN="principal:root;password:af88b69c4f1215a1e62202ceb45f5a92;realm:default-realm;role:ALL"

[eric-maynard]

I think many of these issues should be fixed by:
#30

[AlexMercedCoder]

Just in case anyone want to go through the steps I did get it all going here is write up:
https://www.dremio.com/blog/getting-hands-on-with-polaris-oss-apache-iceberg-and-apache-spark/

[collado-mike]

@AlexMercedCoder can you confirm the linked PR did solve your issue? Can we close this?

[collado-mike]

I'll close for now. Feel free to reopen if this is still an issue