[SPARK-43980][SQL] introducing select * except syntax

[milastdbx]

What changes were proposed in this pull request?

Changing parser to support new syntax when using * to fetch columns from source.
Introducing new expression UnresolvedStarExcept in visit method when newly introduced syntax is created.
Expansion of this expression is core logic of feature.

Why are the changes needed?

Introducing new SELECT * EXCEPT (col1, col2)

Does this PR introduce any user-facing change?

Yes, this PR introduces new SQL syntax, which is used to explicitly exclude columns from star projection.

How was this patch tested?

Unit tests.
Generated new golden files.

Was this patch authored or co-authored using generative AI tooling?

No

[dongjoon-hyun]

Thank you for making a PR, @milastdbx . Could you provide some references for this syntax in other SQL environments?

[milastdbx]

Thank you for making a PR, @milastdbx . Could you provide some references for this syntax in other SQL environments?

When you say other SQL environments, what exactly are you referring to ?

[dongjoon-hyun]

Popular ones like Apache Hive, Apache Flink, Presto, MySQL, PostgreSQL, MySQL, Oracle, Teradata?

[milastdbx]

Popular ones like Apache Hive, Apache Flink, Presto, MySQL, PostgreSQL, MySQL, Oracle, Teradata?

I don't think any other platform supports it. I maybe rushed the comment saying its ansii standard. I'll update PR description

[dongjoon-hyun]

Apache Spark community want to be as compatible as possible other SQL environments in order to avoid potential lock-ins effects. Given that context, this new syntax looks too esoteric to be accepted from my perspective.

Some other Apache Spark committers may have different opinions.

[HyukjinKwon]

Yeah, i wouldn't add this as a dialect but better stick to other DBMSes or ANSI standard

[cloud-fan]

I think this is a useful feature, see https://stackoverflow.com/questions/29095281/select-all-the-columns-of-a-table-except-one-column and https://dba.stackexchange.com/questions/1957/sql-select-all-columns-except-some

In fact, Spark already has a SELECT regex feature: #18023 . We can't ignore the need to flexibly select some but not all columns.

If we have to follow an example, Databricks SQL supports this syntax: https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-qry-select.html

[HyukjinKwon]

im okay if we already have the variant.

[surjikal]

Bigquery supports SELECT * EXCEPT:

• https://cloud.google.com/bigquery/docs/reference/standard-sql/query-syntax#select-except

On the other hand, Snowflake and DuckDB use SELECT * EXCLUDE:

• https://docs.snowflake.com/en/sql-reference/sql/select
• https://duckdb.org/docs/sql/query_syntax/select.html

---

For what it's worth, I do things like df.drop('foo')['*'] very often to achieve a similar thing in dataframe land.

My app's DB (not spark) doesn't support the EXCEPT feature and I wish it did, there's a fair amount of extra SQL generated as a workaround. Definitely a good feature from my perspective.

[cloud-fan]

there is a test failure: SparkThrowableSuite.Error classes are correctly formatted

[cloud-fan]

The error in docker-integration-tests is unrelated, thanks, merging to master!