Python: Add support for providing SSL config for REST Catalog client.

[dhruv-pratap]

Context: https://apache-iceberg.slack.com/archives/C029EE6HQ5D/p1666112648002419

We at Netflix here are trying to integrate PyIceberg 0.1.0 with our Iceberg Rest Catalog Service and realized there is a gap in the PyIceberg rest client that we need to address.
For background, at Netflix all client-server interaction happens over TLS and is client side auth enforced for security purposes. The rest client that sits inside PyIceberg at present uses requests module for interaction with rest catalog service. Although this module allows the CA trust bundle to be set via an environment variable, but it does not allow a similar mechanism for setting client side certificates via environment variable and has to be done programmatically when setting up a requests client.

The below two approaches were discussed on the Slack thread:

1. RestCatalog exposes requests.Session() and accepts a pre-configured Session object during initialization. This puts the onus on the PyIceberg consumer to configure the session with correct auth mechanism, connection pooling, custom headers etc., and makes the RestCatalog client dumb and simply use the provided session to perform API interaction. The current RestSpec kind of assumes/dictates the auth mechanism to be oauth which might not be the case for every enterprise.
2. The other alternative is to define new set of PyIceberg properties and add to the current PyIceberg configuration spec to accept SSL configuration. Example: catalog.rest.ssl.client.key catalog.rest.ssl.client.cert catalog.rest.ssl.ca This would be minimal amount of changes, but I could see this list growing overtime with every new enterprise adoption requiring customization.

After discussion with @samredai and @Fokko we agreed on approach #2, and this PR is to address the same.

[dhruv-pratap]

High level summary of changes done as part of the PR:

1. Rest Catalog encapsulates a requests.Session
2. Configure Session object with SSL config, if provided, during initialization.
3. Configure existing headers one-time in the session object during initialization.
4. Use the one-time initialized Session object for all further REST calls.
5. Add unit tests for SSL config initiation, and enforce HTTP header checks in existing REST mock tests.
6. Add documentation for REST Catalog client mTLS configuration.

[Fokko]

Thanks for working on this @dhruv-pratap! I like it a lot. Another approach we could also take is to have a function _create_session() -> Session that will set up the session. This would both set the SSL and the headers, and in the constructor, we would do:

self.session = self._create_session()

This way we have a single place where we set up the session, instead of two methods. WDYT?

[Fokko]

I like this a lot, much nicer than having to pass in the headers every time.

[dhruv-pratap]

Thanks for working on this @dhruv-pratap! I like it a lot. Another approach we could also take is to have a function _create_session() -> Session that will set up the session. This would both set the SSL and the headers, and in the constructor, we would do:

self.session = self._create_session()

This way we have a single place where we set up the session, instead of two methods. WDYT?

I did think about that initially, but the self._fetch_access_token(credential) REST call poses a kind-of chicken-and-egg problem. You need the session SSL config to be configured to make that REST call, and then the session's header has to be enriched with the Auth Token. Let me try to give it a shot to see what that looks like and send an amendment.

[dhruv-pratap]

Suggested changes pushed in recent commit. Please review.

[Fokko]

@dhruv-pratapI didn't think of the chicken-and-egg problem. The current version isn't working:

RESTError: HttpMediaTypeNotSupportedException: Content type 'application/json' not supported

It already sets the application/json header, but we need to set the application/x-www-form-urlencoded header for the signer:
https://github.com/apache/iceberg/blob/master/python/pyiceberg/catalog/rest.py#L237

We either need to set the headers after we fetch the token, or revert to the previous version.

[dhruv-pratap]

Sorry, I didn't have an OAuth environment to integration test it against.

Changed the order in recent commit, now setting the HTTP headers after fetching the OAuth token. Also, enforcing the content-type check to application/x-www-form-urlencoded for OAuth calls in unit tests.

[Fokko]

No problem at all. I've just checked and it works 👍🏻

[Fokko]

This is awesome, thanks @dhruv-pratap 👍🏻