[SPARK-41290][SQL] Support GENERATED ALWAYS AS expressions for columns in create/replace table statements

[allisonport-db]

What changes were proposed in this pull request?

Enables creating generated columns in CREATE/REPLACE TABLE statements by specifying a generation expression for a column with GENERATED ALWAYS AS expr.

For example the following will be supported:

CREATE TABLE default.example (
    time TIMESTAMP,
    date DATE GENERATED ALWAYS AS (CAST(time AS DATE))
)

To be more specific this PR

1. Adds parser support for GENERATED ALWAYS AS expr in create/replace table statements. Generation expressions are temporarily stored in the field's metadata, and then are parsed/verified in DataSourceV2Strategy and used to instantiate v2 Column.
2. Adds TableCatalog::capabilities() and TableCatalogCapability.SUPPORTS_CREATE_TABLE_WITH_GENERATED_COLUMNS This will be used to determine whether to allow specifying generation expressions or whether to throw an exception.

Why are the changes needed?

GENERATED ALWAYS AS is SQL standard. These changes will allow defining generated columns in create/replace table statements in Spark SQL.

Does this PR introduce any user-facing change?

Using GENERATED ALWAYS AS expr in CREATE/REPLACE table statements will no longer throw a parsing error. When used with a supporting table catalog the query should progress, when used with a nonsupporting catalog there will be an analysis exception.

Previous behavior:

spark-sql> CREATE TABLE default.example (
         >     time TIMESTAMP,
         >     date DATE GENERATED ALWAYS AS (CAST(time AS DATE))
         > )
         > ;
Error in query: 
Syntax error at or near 'GENERATED'(line 3, pos 14)

== SQL ==
CREATE TABLE default.example (
    time TIMESTAMP,
    date DATE GENERATED ALWAYS AS (CAST(time AS DATE))
--------------^^^
)

New behavior:

AnalysisException: [UNSUPPORTED_FEATURE.TABLE_OPERATION] The feature is not supported: Table `my_tab` does not 
support creating generated columns with GENERATED ALWAYS AS expressions. Please check the current catalog and 
namespace to make sure the qualified table name is expected, and also check the catalog implementation which is 
configured by "spark.sql.catalog".

How was this patch tested?

Adds unit tests

[AmplabJenkins]

Can one of the admins verify this patch?

[cloud-fan]

Does the SQL standard say anything about the restrictions of the generate expression? Can we allow GENERATED AS rand()? I think at least catalyst should have a rule to check the expression and make sure the data type is the same as column type.

[cloud-fan]

also cc @huaxingao @sunchao

[zsxwing]

We can just call this method supportsGeneratedColumns, so that we don't need to add a new API if we want to extend generated columns in other statements in future.

[allisonport-db]

Couldn't this be an issue though if we add the syntax to another statement (for example alter table add column) and the catalog/data source (whatever we land on) hasn't updated their implementation yet to accomodate for it?

[allisonport-db]

Does the SQL standard say anything about the restrictions of the generate expression? Can we allow GENERATED AS rand()? I think at least catalyst should have a rule to check the expression and make sure the data type is the same as column type.

Checking for data type make sense.

Other restrictions we can enforce

• no UDFs
• only deterministic functions
• no subqueries
• no window functions
• no aggregate functions
• no generator functions
• interdependence with other generated columns

[dongjoon-hyun]

Thank you for update. Only one question about error method remains.
+1, LGTM.

[viirya]

This can be more specified because it is a V1 table?

[allisonport-db]

I can add a new error class for it if you think it's necessary?

[viirya]

Looks good for backport.

[cloud-fan]

Are we going to document this new SQL syntax somewhere

This is a long-standing problem. There are quite some features that only have DS v2 API but no implementation, e.g. UPDATE, MERGE, DELETE. I think we should make the testing in-memory catalog/table a builtin connector, so that we can document these SQL syntaxes with valid examples. cc @huaxingao

[dtenedor]

Thanks for adding the extra test cases! If we're banning subquery expressions, the validation checks become much simpler now.

[cloud-fan]

thanks, merging to master/3.4!