[SPARK-38795][SQL] Support INSERT INTO user specified column lists with DEFAULT values

[dtenedor]

What changes were proposed in this pull request?

Support INSERT INTO commands with user specified column lists with DEFAULT values.

For example:

CREATE TABLE t (x INT DEFAULT 42, y STRING DEFAULT "abc") USING PARQUET;
INSERT INTO t (y) VALUES ("def");
SELECT x, y FROM t;
> 42, "def"

Why are the changes needed?

This PR allows INSERT INTO commands with explicit user specified column lists to benefit from DEFAULT column support, in addition to those without. This expands the functionality provided by this feature.

Does this PR introduce any user-facing change?

Yes, more types of INSERT INTO commands may use DEFAULT values.

How was this patch tested?

This PR adds additional unit test coverage.

[gengliangwang]

@dtenedor I was focusing on other works today. Will take another look tomorrow.

[dtenedor]

@dtenedor I was focusing on other works today. Will take another look tomorrow.

That's totally OK! You are finishing up the code sync conductors shift and I am starting the next one today. Thanks again for all your careful review and help.

[AmplabJenkins]

Can one of the admins verify this patch?

[dtenedor]

Thank you for your review! Updated according to excellent suggestions, please take another look.

[gengliangwang]

LGTM except a few minor comments.

[dtenedor]

Thanks again for your review! This should be ready to merge at your convenience.

[gengliangwang]

LGTM, thanks for the work!

[gengliangwang]

Merging to master