[SPARK-38707][SQL] Allow user to insert into only certain columns of a table

[morvenhuang]

What changes were proposed in this pull request?

Update the column number check logic for INSERT INTO statement, avoid checking all columns of the table when user has specified a column list in the statement.

Why are the changes needed?

When running INSERT INTO statement, it's quite common that user wants to insert only certain columns, especially when the rest columns have default value.

Currently, spark allows user to specify column list for INSERT INTO statement only when the column list contains all columns of the table. If user does not provided a completed list of column, it will result in an AnalysisException.

This patch allows user to insert into only certain columns of the table, which will help when excuting INSERT INTO, especially when excuting them on RDBMS.

Does this PR introduce any user-facing change?

Yes.

How was this patch tested?

New test case added.

[AmplabJenkins]

Can one of the admins verify this patch?

[morvenhuang]

Can one of the admins verify this patch?

There're some failed test, I'm looking into it.

[HyukjinKwon]

@morvenhuang it would be great to assess this further if you add some references of other DBMSes that support this syntax. Does Hive support this?

[morvenhuang]

@HyukjinKwon Hi Hyukjin, thank you for your time. I believe this is a SQL-92 standard, most RDBMSes I've known support inserting only certain columns of a table, like MySQL/Oracle/MS SQL SERVER/Teradata. And yes, Hive also supports this.

[HyukjinKwon]

cc @dtenedor FYI. Seems like actually it's already implemented with spark.sql.defaultColumn.useNullsForMissingDefautValues configuration?

[morvenhuang]

@HyukjinKwon Hi Hyukjin, thanks for the comment, that implementation is a great job, although there's a tiny issue due to unnecessary column number check, say we have a table t1 with 2 columns c1 int, c2 int, insert into t1(c1) values(100) is still gonna fail even when the useNullsForMissingDefautValues is enabled.

I believe that column number check is not necessary anymore when this option is enabled, since all omitted column(s) will be added back to query automatically during parse, which means insert into t1(c1) values(100) is actually insert into t1(c1, c2) values(100, null) after parse.

I've made a commit to get rid of the check, so that the insert into t1(c1) values(100) statement can work.

[morvenhuang]

@HyukjinKwon Hi Hyukjin, could you please help to verify this? Many thanks.

[dtenedor]

FYI I also have this relevant PR for developing INSERT INTO support out for review as well :)

#36077

[morvenhuang]

@dtenedor that's great, thank you for pointing out, I'm gonna close this one.

[dtenedor]

@morvenhuang looking forward to working more on Apache Spark with you and the community!