HiveCatalog: add method to get current metadata location for a given table identifier

[henrib]

Feature Request / Improvement

WHAT
Add a method to get current the metadata location for a given table identifier.
WHY
To verify if a Table object points to the latest version known, the path to the actual metadata file location is the source of truth. This is useful for server-side caching to detect stale versions and/or implement an eTag/conditional reload (by digesting location to create the eTag for instance).
HOW
Adding that method to HiveCatalog that queries the underlying HMS Store; if the table does not exist or is not an Iceberg table, return null otherwise the metadata file location.

Query engine

Hive

Willingness to contribute

• I can contribute this improvement/feature independently
• I would be willing to contribute this improvement/feature with guidance from the Iceberg community
• I cannot contribute this improvement/feature at this time

[henrib]

Pull request at #13801

[RussellSpitzer]

Can you explain a little more about the usages here? I assume this for a system which is using a HiveCatalog and wants to check the current state of the table, but doesn't want to check the HMS directly? I'm mostly trying to see how widely required this would be since it's essentially just a shortcut for client.loadTable().property("metadata_location")

Mostly my thoughts here are if we want to move away from Required metadata.json this is a bit of a step in the other direction but since it would only be for HiveCatalog it may be fine since it's specific to that implementation

[henrib]

It is intended to be used within HMS when exposing a REST Catalog; the catalog instance is wrapped in a a cache that delegates all calls besides loadTable. On a cache hit, the Table object staleness is assessed by looking at the latest known metadata location from the HMS store. This ensures that loadTable() always returns the latest committed version.
This could be performed directly but the HiveCatalog has all the method - and a client pool - that makes it easy to implement and logical to expose.
The goal is definitely not to be a shortcut of client.loadTable().property("metadata_location"); it is intended to be a simple (and hopefully fast) query avoiding materializing the object that only makes sense within HiveCatalog.

[RussellSpitzer]

I think that makes a lot of sense, especially in the context of wrapping the HiveCatalog with a RestAdapter

[pvary]

I think, the same could be true for other catalog implementations as well. That is why I proposed to add the new method to the Catalog API.

[henrib]

Although this may be useful to other catalog implementations, it should not be a mandatory method - as a method of the Catalog interface. Adding a default method implementation that performs client.loadTable().property("metadata_location") has very low usefulness as a a one-liner that goes against the functional intent (fast access to last known metadata location). And other catalog implementations might have different and more efficient ways of assessing staleness. From that perspective, it does not look like a good addition to the Catalog API.
And if it is determined later that this should be generalized, the signature 'public String getTableMetadataLocation(TableIdentifier identifier)' can always be added to the Catalog interface. In the current state, I believe #13801 should remain specific to HiveCatalog.

[pvary]

I think most of the other catalogs could also benefit from a method which provides a cheap implementation for the etag generation without loading the table metadata itself

[gaborkaszab]

Thanks for raising this, @henrib ! The algorithm you described above more or less resonates with the proposal I have for freshness-aware loading for REST catalog. I also had it in mind to investigate if the very same approach could be used for HiveCatalog too so thanks again for bringing this up!

Where the REST-specific proposal deviates from yours are the following:

1. What to use to describe particular state of a table
   For REST we agreed to use ETags that are returned by the REST server and is opaque (can't be assumed that it's current metadata location). Based on your description, for HiveCatalog you'd be more direct and explicitly use metadata location as 'identifiers' for state.
2. Level of responsibilities
   What I mean here is that for the freshness-aware loading implementation for REST catalog is performed "seamlessly" and doesn't require any extra info to be exposed to the clients of the catalog. While the proposal you have would expose a metadata location and would leave the rest of the work to the clients.
3. Number of catalog interactions
   The REST catalog proposal makes a single call to the REST catalog regardless if the table has been changed or not. With your proposal there is one call if the table hasn't changed or two if it has.

I also checked the referenced Hive change and I have the impression the only usage of this exposed metadata location is going to be from the Hive/HMS code and it won't be leveraged from the Iceberg code, right? We can't expect a follow-up change to implement freshness-aware loading in HiveCatalog, right? It won't give any short term help for you but let me just brainstorm here:
Since REST catalog will have it's own built-in, seamless freshness-aware loading implementation, I think we can gather the building blocks what it would require to have the same for HiveCatalog too. That way there won't be any need for the clients like HMS, Hive etc. to implement their own way of caching and refreshing. I myself aren't much familiar with HMS API (will check later on as homework), but I think the key would be to have a load table API (similarly to REST) where HMS can add an ETag to the result, and then the client can attach this ETag to the calls after that. Also this same API should give an 304-Not Modified kind of error in a way to indicate to the client that the table hasn't changed since the ETag the client sent.
This, I'm pretty sure would take some time to implement, release and have available in Iceberg, so I guess the question is that can HMS in the meantime live with each loadTable requests resulting a table load (and not just a metadata location load). I guess there are scenarios already (e.g. streaming ingestion) where the proposed Hive/HMS approach would simply got things worse (by getting location first and then load table second, because of the frequent changes).

Sorry for the long message. LMK WDYT!

[gaborkaszab]

An additional thought: This exposed metadata location would be used in HMS code if I'm not mistaken, right? Why use HiveCatalog API to go to HMS to get the full table and then get a single property from that table then? Isn't it feasible to directly read into HMS for that table property (I recall metadata location is a table property in HMS) avoiding going through the HiveCatalog interface and loading the whole table? This would also buy us some time until we figure out how to implement freshness-aware loading in HiveCatalog.
Let me know what I miss here :)

[henrib]

About using the HiveCatalog API to get to the Hive table, the code is simple and - hopefully - the performance gain that we could get by creating a new dedicated call (thrift and all) and/or direct SQL is a dwarf compared to not caching. But it can certainly be done.
Again, this new method is only meant to improve the HMS / HiveCatalog server side cache by avoiding serving stale entries.
There are certainly other cases where 'staleness detection' would be useful; however, my initial issue & PR is simple and narrow, just one method in a specific Hive only class.
I believe the wider and complex issues that are now discussed deserve their own issue and discussions.

[pvary]

@henrib: Could you please elaborate a bit on this please?

Again, this new method is only meant to improve the HMS / HiveCatalog server side cache

Are you referring to HS2 side cache? I'm asking, because from the HiveCatalog perspective it is a client side, and I would like to understand the case correctly.

Maybe adding the cache to the HiveCatalog would be better. In this case we would not need to expose additional methods, and archive the same functionality.

[henrib]

@pvary Hive 4.1 MetaStore server exposes the Iceberg REST Catalog API through a dedicated servlet; that servlet delegates its calls to a HiveCatalog instance wrapped in a cache to try and avoid reloading tables.
I understand the cache was initially meant for client-side; however, besides loadTable(), it delegates every call to a catalog instance. By overloading loadTable(), it allows checking that the table object to be returned is not stale (this replaces for the better a TTL based eviction).
We could add the cache to HiveCatalog itself; we can probably 'drive' its existence through configuration.
(for ref, apache/hive#6022 is the attempt at fixing the cache in HiveMetastore)