Cannot Drop Table Created with HiveIcebergStorageHandler Enabled but Metadata.json is Missing

[RussellSpitzer]

If a user is working with the HiveIcebergStorageHandler then they can get into a bit of a tricky situation. Say perhaps, that they delete their metadata.json or alter that property such that it no longer points to a valid file. Any subsequent DDL or operations will fail because for Hive to modify the table it must first load metadata.json. So if the file is dropped the the table is undroppable. This may also be an issue in case the table property is changed to point to a bad metadata.json path.

I'm not sure if this is something we can guard against but I thought I'd make a note of it

[RussellSpitzer]

@marton-bod + @pvary Just in case you had some thoughts

[marton-bod]

This is a good point, and I think we can and should guard against that. The table currently becomes undroppable only because we try to load the Iceberg table prior to dropping the HMS table. We do this because Hive2 and Hive3 do not clean up the metadata folder completely so we need to cache the io to perform the deletion later.

What we can do instead is try to load the Iceberg table as it is, but if it fails, just log a warning message that the table was unloadable for whatever reason, but otherwise proceed with the HMS table drop. This might leave some undeleted metadata files behind but it's still much better than leaving the table undroppable.

[pvary]

@RussellSpitzer: How likely is this situation based on your experience?

In Hive we faced similar situation a few times before, for example:

• Dropping the table and failing to remove the files

Sadly we still do not have a way in Hive to return a warning message along with reporting the success of the drop table operation. What we did in these cases:

• Log a warn message in the logs, and silently (for the user) drop the metadata and skip the file removal.

This looks like the same solution suggested by @marton-bod, and I think it is much easier to recover from a problem if we follow this way. It is easier to drop the remaining data from the Iceberg table later, than recreating the appropriate files just so the metadata could be deleted as well.

Thanks,
Peter

[RussellSpitzer]

So far this has never cropped up intentionally, but I had a user who saw the hive.engine.enabled property and set it on a large number of multiuser tables.

This caused a bit of havoc. The deletion issue then happened first as they were used to standard hive tables and removed the data files from some tables before the dropping them from the catalog.

The next issue we kept hitting is that there were no iceberg jars on the hive class-path which is a problem because you can always set the storage handler, but once set, you can't remove the property without the jars on the class-path

[RussellSpitzer]

So for now it's 1/1 with users who set this property 😊. They were setting up a framework for tons of other users and use cases so they created a lot of entities

[marton-bod]

We do have the post-drop hook available though to convey information to the user after-the-fact, so for example we can throw an exception to draw the user's attention to this anomaly, despite the success of the core DROP operation.

So, I see two options:

• we try to load the Iceberg table in the pre-drop hook, save in a boolean whether it succeeded, DROP the table as per Hive, and if the load failed earlier then throw a descriptive exception in the post-drop hook explaining what happened (i.e. the drop succeeded, but there was this anomaly, so please look into it, there might metadata files undeleted which might need manual intervention, etc.)
• we try to load the Iceberg table in the pre-drop hook, log a descriptive error msg if it fails, DROP the table as per Hive, nothing else (i.e. what I described a couple days ago)

I would prefer the first one, since I think it's more explicit (the drawback is that it fails the operation from a return code point of view, but not from a functional pov). The second one is silent and we can't expect the user to inspect the logs after each DROP operation looking for warning signs.

[pvary]

@marton-bod: I would prefer your first suggestion:

• Log a warn message
• DROP TABLE should be successful

There are automatic clients who might expect the table to remain if the DROP TABLE command is not successful. This could be confusing. We either do not drop a table metadata and return an error, or drop the table metadata and return successfully. At least that was the reasoning behind our decision in the aforementioned case.

The next issue we kept hitting is that there were no iceberg jars on the hive class-path which is a problem because you can always set the storage handler, but once set, you can't remove the property without the jars on the class-path

This is another story might worth another issue, if we are not able to drop the table even from Spark, or Java clients

[marton-bod]

.. who might expect the table to remain if the DROP TABLE command is not successful

This is a good point! Let's go with the first idea then - I'll put together a PR soon.

[RussellSpitzer]

Made #2578 for the engine enabled issue no jars on CP @pvary, Thanks ya'll for helping out with this

[fcomuniz]

Having the issue on 0.12.0 using spark to drop the tables with Hive Meta Store, is there a way to do it without spark?

[KarlManong]

Having the issue on 0.12.0 using spark to drop the tables with Hive Meta Store, is there a way to do it without spark?

We got the same issue. I drop the table in Hive Service. That's the only way I know. Also I deleted the data files manually.

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.