Add properties support for HadoopTables.load()

[qqchang2nd]

Feature Request / Improvement

Description:
Currently, HadoopTables.load() doesn't support passing custom properties when loading tables. While HiveCatalog and HadoopCatalog support manifest caching through their initialize() method (as implemented in #4518), HadoopTables lacks this capability. This enhancement adds property support to HadoopTables.load() to enable manifest caching and other configurations.

Problem:

• HadoopTables lacks the ability to configure manifest caching during table loading
• Unlike HiveCatalog and HadoopCatalog which can enable manifest caching through initialize(), HadoopTables has no mechanism to pass these settings
• This creates inconsistency in how manifest caching can be configured across different catalog implementations

Query engine

None

Willingness to contribute

• I can contribute this improvement/feature independently
• I would be willing to contribute this improvement/feature with guidance from the Iceberg community
• I cannot contribute this improvement/feature at this time

[RussellSpitzer]

HadoopTables isn't a catalog though? I always thought of it more as a low level method for loading up an Iceberg table without a catalog being present. Can you explain the use case a bit more here?

[qqchang2nd]

You're right that HadoopTables isn't a catalog - it's a lower-level implementation for managing Iceberg tables directly on HDFS without a catalog.

Let me explain our use case:
We have our own metadata management platform internally, and during our initial evaluation of Iceberg, we chose HadoopTables for table creation and management without fully investigating the differences between HadoopTables and catalogs.
After running in production for a while, we noticed that some customer environments were experiencing slow performance during sql-analyze operations. Investigation revealed this was because tables created via HadoopTables don't support manifest caching.

To address this, I modified the Iceberg source code to allow HadoopTables.load() to accept properties, enabling manifest caching support which significantly improves sql-analyze performance.

[RussellSpitzer]

I'm very interested to know how manifest caching is helping you, To my knowledge it's generally disabled for just about everyone. Do you have a long lived processes that just repeatedly use independent HadoopTables instances?

[qqchang2nd]

Let me share our use case at Sensors Data. We use HadoopTables to manage Iceberg tables with Impala as our query engine.

In one of our customer environments, we encountered a query that only accessed one day's worth of data but took about 12 seconds total, with 6 seconds spent just on plan analysis. Through Arthas tracing, we identified the bottleneck was in manifest loading. After enabling manifest caching, the plan analysis time was reduced to under 200ms. For these types of queries, manifest caching can improve overall query performance by up to 2x.

At Sensors Data, we currently have over 100 customers using the HadoopTables management approach, and this number is expected to grow to thousands in the future. We don't have plans to migrate to other catalog implementations like HadoopCatalog.

The performance improvement from manifest caching is significant for our use case because:

1. We have long-running Impala daemons that repeatedly access these tables
2. The manifest files remain relatively stable during query execution
3. The memory overhead of caching is acceptable given the performance benefits

Would you be interested in more detailed performance data or our specific usage patterns?

[RussellSpitzer]

So you have a long term process that uses HadoopTables that end users access? That's what I was asking about, in what context are you using this API.

[qqchang2nd]

In Impala, the catalogd daemon maintains metadata caching. Within catalogd, the classes IcebergHadoopCatalog, IcebergHiveCatalog, and IcebergHadoopTables are wrappers around Iceberg's internal HadoopCatalog, HiveCatalog, and HadoopTables implementations respectively.

As referenced in IMPALA-11658, manifest caching properties are passed to IcebergHadoopCatalog and IcebergHiveCatalog through Catalog.initialize(). Building on this approach, I modified IcebergHadoopTables to pass manifest caching properties through HadoopTables.load() to achieve the same caching benefits.

So yes, we have a long-running process (Impala's catalogd) that uses HadoopTables, and enabling manifest caching in this context provides significant performance improvements for our users' queries.

[RussellSpitzer]

I think my biggest concern here is that it's not a generally useful change an opens up a lot of parameterization for something we generally do not want to be used. My gut here would be to say that this should just be a Impala internal change and not something in the general library.

[qqchang2nd]

I understand your concern about the potential for increased parameterization and the implications it may have. However, I believe the changes I've made are quite generalizable and beneficial.

The modification introduces an interface that allows for passing properties while still preserving the original usage method. The primary issue with Iceberg's current implementation is that HadoopTables effectively hides the ability to pass properties to HadoopFileIO. This limitation restricts its flexibility.

By enabling the ability to pass properties to HadoopTables, we can enhance its functionality without compromising the simplicity of the original API. This change not only addresses our specific use case but also opens up possibilities for other users who may benefit from similar optimizations.

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'