Return actual expiry including token expiry for presigned URLs

[arielshaqed]

Clients can use our API to request presigned URLs to underlying storage. The lifetime of this URL is given by the minimum of:

• The presigned expiry time, configured for the block adaptor.
• The expiry time of any authentication tokens.
  For instance, when lakeFS is configured to run using AWS IAM STS, presigned URLs will expire when the current token expires. The client has no way of knowing either of these times. While the expiry time is static and conceivably might be configured on the client as well, lakeFS auth token expiration is dynamic and cannot be guessed by the client.

This might currently be causing isses at user sites using lakeFS Delta Sharing with Unity.

Add an expiration field to presigned responses.

[arielshaqed]

First priority currently AWS (sorry, Azure folks...).

[johnnyaug]

Not sure I agree with this solution. I would argue that it's within the responsibility of the lakeFS admin to make sure that lakeFS authenticates properly with the storage. Returning an expiry time which is deduced from two different expiry times seems confusing, and it's not really clear what the user is meant to do with this information.

We could help in other ways. For example, by introducing a mechanism to renew the storage session when it's about to expire.

@arielshaqed, could you elaborate what issue this may be causing on the user side?

[arielshaqed]

@arielshaqed, could you elaborate what issue this may be causing on the user side?

Sure!

lakeFS Delta Sharing has to return presigned URLs. Meanwhile lakeFS Cloud authenticates to AWS S3 using some weird K8s/EKS token-based scheme -- that uses a token that expires. So we need to specify on the protocol when the presigned URL expires, otherwise the Delta Sharing client can try to use an expired URL and get a nasty error. This already happened.