[SPARK-51728][SQL] Add SELECT EXCEPT Support

[Gschiavon]

What changes were proposed in this pull request?

Jira - https://issues.apache.org/jira/browse/SPARK-51728

This change introduces support for the SELECT * EXCEPT(col1, col2, ...) syntax, which allows users to project all columns from a dataset except for a specified subset. This pattern is especially useful in wide tables where explicitly listing all desired columns would be verbose and error-prone.

The syntax is already widely adopted in other SQL engines like BigQuery (docs), and adding support in Spark SQL helps improve compatibility, reduce boilerplate, and improve developer ergonomics when working with large schemas.

Why are the changes needed?

I think SELECT EXCEPT it's useful when working with large schemas and other DBMSes have it (e.g BigQuery) so I think it'd be good to add support in Spark as well

Does this PR introduce any user-facing change?

It doesn't include changes as it'd be a new feature

How was this patch tested?

It was tested in SQLQuerySuite by adding a dedicated test to SELECT EXCEPT

Was this patch authored or co-authored using generative AI tooling?

No

[ik8]

This is already implemented here:

When spark.sql.parser.quotedRegexColumnNames is true, 
quoted identifiers (using backticks) in SELECT statement are interpreted as 
regular expressions and SELECT statement can take regex-based column specification.
For example, below SQL will only take column c:
SELECT `(a|b)?+.+` FROM (SELECT 1 as a, 2 as b, 3 as c)

[Gschiavon]

This is already implemented here:

When spark.sql.parser.quotedRegexColumnNames is true, 
quoted identifiers (using backticks) in SELECT statement are interpreted as 
regular expressions and SELECT statement can take regex-based column specification.
For example, below SQL will only take column c:
SELECT `(a|b)?+.+` FROM (SELECT 1 as a, 2 as b, 3 as c)

@ik8 I don’t think this is the same. Imagine that you have a dataset with 200 columns and you want to select all but 20 columns, how would you do that? It would look really complex.

I think select except is different from select regex

[ik8]

This is already implemented here:

When spark.sql.parser.quotedRegexColumnNames is true, 
quoted identifiers (using backticks) in SELECT statement are interpreted as 
regular expressions and SELECT statement can take regex-based column specification.
For example, below SQL will only take column c:
SELECT `(a|b)?+.+` FROM (SELECT 1 as a, 2 as b, 3 as c)

@ik8 I don’t think this is the same. Imagine that you have a dataset with 200 columns and you want to select all but 20 columns, how would you do that? It would look really complex.

I think select except is different from select regex

@Gschiavon either way you will have to specify the columns you don't want to select.
I think this is a good option to have(your pr with the EXCEPT), just wanted to let you know that it is already supported.