Add getting-started example with external authentication

[adutra]

A new getting-started docker-compose example using external authentication with Keycloak.

[singhpk234]

would be really helpful if we can add a bit about what the custom scopes catalog / sign would mean for Polaris

[adutra]

It means nothing to Polaris. Scopes are between the client and Keycloak. I think I should remove those scopes.

[aacampbell]

The confusion might come from the fact that when making a request to the internal deprecated iceberg oauth token endpoint we include "scope": "PRINCIPAL_ROLE:ALL" and it's not clear why - especially when we don't include it in the request to keycloak. In fact I don't think the documentation mentions at all how these role work, which to request, and why. I think there might even by a typo in the external-idp document where it sometimes says POLARIS_ROLE: instead of PRINCIPAL_ROLE:. A lot of behaviour requires delving into source code to learn.

[adutra]

Yes I agree that internal authentication looks a bit mysterious to users right now.

The internal authentication retrieves its principal roles from the OAuth2 scope field, and expects scopes in the form PRINCIPAL_ROLE:<role_name>. The scope PRINCIPAL_ROLE:ALL is a pseudo-role: it means the client is requesting all of the principal's roles in the system.

I will provide some clarification to the docs in a separate PRs. Thanks for bearing with us 😄

[singhpk234]

maybe :

Suggested change

	actual principal ID, name and roles of the user. Note that principals must have been created in Polaris beforehand.
	actual principal ID, name and roles of the user. Note that principals must have been created in Polaris beforehand and principal_id etc should correspond to principal info in polaris

[singhpk234]

may be minor doc about which one is checked first helpful or can link existing doc for external idp, wdyt ?

[aacampbell]

Yeah, I think it would be really helpful to put some of the information added here into the existing external-idp document, and to put a link into that existing document to this one. In particular, making it very clear in the external-idp document that principals must already exist in Polaris - I got my hopes up about full external identity control, but have since found the milestone documents about user federation.

[adutra]

Yes, for now you need to create the users in both systems. This is admittedly impractical. We still need to implement full identity federation. I am working on this.

[singhpk234]

SCIM support ?

[aacampbell]

This new example is very helpful, and very clearly shows how things work. One thing to note: The create_catalog.sh script doesn't work except for the first realm, because it's hardcoded to hit the deprecated iceberg token endpoint. Switching the URL to Keycloak's, and dropping the scope argument, looks like it'll work.

It does lead to an interesting train of thought. If we can expect that the deprecated iceberg catalog/v1/oauth/tokens endpoint will eventually be removed, what will be best practice? We will all have to use the external authentication type, which means complete dependence on an external system to use even the bootstrap admin user. I see there's another Issue #2238 regarding the bootstrapped admin having a hardcoded root principal name, which is what the principal-mapper must match. This would require a special-case external user in order to do any administration or even setup of other principals in Polaris. Being able to choose the bootstrapped admin's principal name would go some way to allowing uniform mapping of external IdP user attributes to principal names in Polaris, but this is getting to be more of an external concern. What isn't solved is the external dependence for doing any administrative actions. Perhaps a small/basic token endpoint in the management api for getting administrative tokens? That might prevent Polaris from doing too much of the lifting for Auth while still allowing it to be securely administered without external dependencies. I think this would effectively be the same as the current "mixed" authentication case, except the endpoint is under the management api rather than the Iceberg catalog api.

I feel like I got pretty far off topic here. Is there a more appropriate forum?

[adutra]

One thing to note: The create_catalog.sh script doesn't work except for the first realm, because it's hardcoded to hit the deprecated iceberg token endpoint. Switching the URL to Keycloak's, and dropping the scope argument, looks like it'll work.

With this PR, you can now pass your own token to the script.

It does lead to an interesting train of thought. If we can expect that the deprecated iceberg catalog/v1/oauth/tokens endpoint will eventually be removed, what will be best practice? We will all have to use the external authentication type, which means complete dependence on an external system to use even the bootstrap admin user. [..] I feel like I got pretty far off topic here. Is there a more appropriate forum?

Wow, there is a lot in there, thanks for sharing your thoughts.

I would recommend reading this (old) document first for getting an idea of where the problems are, especially the sections about SCIM support and persisting federated entities:

https://docs.google.com/document/d/15_3ZiRB6Lhzw0nxij341QUdxEIyFGTrI9_18bFIyJVo/edit?tab=t.0#heading=h.cu1a1acu4lc5

As you can see, all of this is still TBD and for various reasons, the work there has stalled.

I am preparing a PR to simplify roles validation and extract a few interfaces from concrete classes; this would facilitate the introduction of fully-federated principals and principal roles. But the issue with persisting vs not persisting such entities is still up for debate.

\cc @dimas-b @collado-mike @dennishuo

[singhpk234]

This mostly looks good to me ! thank you @adutra

As final step i was trying to run the getting started my self to see if it works for me, and i am getting some failure (i commented where i see them and whats the message i see), may be its an issue in my system but would really appreciate if you can take a look ! if its not an issue please feel free to proceed !

[singhpk234]

SCIM support ?

[singhpk234]

Thank you @adutra !

just tested the getting-started with key-cloak ! got 200 !