Cannot set metadata.description

[NielsZeilemaker]

Hi, I'm trying to figure out how to set the description of a delta table. Table.comment is used for non-delta tables (https://github.com/delta-io/delta/blob/master/src/main/scala/org/apache/spark/sql/delta/commands/DescribeDeltaDetailsCommandBase.scala#L126), but for delta tables it uses metadata.description (https://github.com/delta-io/delta/blob/master/src/main/scala/org/apache/spark/sql/delta/commands/DescribeDeltaDetailsCommandBase.scala#L170).

However, it seems impossible to set metadata.description. Or am I missing something?

[marmbrus]

Hmm, you are probably right. We should add DDL that lets you set the table description.

[NielsZeilemaker]

I can actually add a comment to a table while creating a delta table. So maybe we can implement a fallback to table.comment if the metadata.description is not set

[marmbrus]

I can actually add a comment to a table while creating a delta table.

This is probably because we don't yet really integrate with the metastore (this is coming soon #85), so you are just getting the default Spark behavior for create table.

We could implement a fallback, but in general we try to keep the transaction log as the source of truth with the metastore only being a "symlink" to the table. Some users access the same table from multiple systems / metastores and it might be confusing if they diverge.

[NielsZeilemaker]

@marmbrus is this something I can help you with? Seems like a small issue I could probably implement/fix myself.

[marmbrus]

Yeah sure! The first question I would look at is if Spark has an extension point here that would allow us to intercept existing DDL to set the comment for a table. @brkyvz might know.

[brkyvz]

With Spark 3.0, there is a new COMMENT ON syntax. You should be able to do it with COMMENT ON TABLE x IS 'my comment' once Spark 3.0 is released