CASCADE WITH Drop Namespace Gives exception

[vinitamaloo-asu]

Apache Iceberg version

1.3.1 (latest release)

Query engine

Spark

Please describe the bug 🐞

Running this command with:
spark.sql(DROP DATABASE IF EXISTS dbname CASCADE)

Gives the below exception:
org.apache.iceberg.exceptions.NamespaceNotEmptyException: Namespace dbname is not empty. One or more tables exist. at org.apache.iceberg.hive.HiveCatalog.dropNamespace(HiveCatalog.java:353) at org.apache.iceberg.spark.SparkCatalog.dropNamespace(SparkCatalog.java:447) at org.apache.spark.sql.execution.datasources.v2.DropNamespaceExec.run(DropNamespaceExec.scala:52) at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:40) at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:40) at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.executeCollect(V2CommandExec.scala:46)

The expectation is that with CASCADE specified, command should delete all tables and the db itself.

[amogh-jahagirdar]

@vinitamaloo-asu this is related to this PR #7275 . Note that cascade would only be supported in Spark 3.3+ since that's when it was added in Spark. The PR should fix the issue (although we'll need to remember to add it in Spark 3.4 as well).

For context, currently SparkCatalog just ignores the passed in cascade flag, but the PR addresses that and the handling in the underlying catalog implementations (in your case HiveCatalog). cc @abmo-x

[vinitamaloo-asu]

Thanks @amogh-jahagirdar for your response. Is it possible to backport it to version < 3.3? I can raise a PR for this if needed.

[amogh-jahagirdar]

@vinitamaloo-asu That would be something to bring up in the Spark community, I don't think there's really anything that can be done in Iceberg for this. The Iceberg SparkCatalog implements certain Spark Dsv2 catalog interfaces, the relevant one in this case being SupportsNamespaces. SupportsNamespaces with a dropNamespace with a cascade option was added in Spark 3.3.
For example in Spark 3.2 you can see https://github.com/apache/spark/blob/v3.2.4/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/SupportsNamespaces.java#L149 that dropNamespace doesn't accept a cascade option.

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'