Run Spark tests with Comet native writer

[comphead]

What is the problem the feature request solves?

Comet native writer works for basic ops, however to move forward and keep compatibility with Spark we need to run Spark writer and partition discovery tests with Comet writer enabled first to make sure we have a solid reliable base for Comet native writer changes

Describe the potential solution

No response

Additional context

No response

[coderfender]

Looking into this

[coderfender]

Updated Spark 4 diff file (ignoring tests) .

[coderfender]

take

[coderfender]

Ignored Tests for Comet Native Parquet Writer

  Common across all Spark versions (3.4, 3.5, 4.0) :

  1. empty file should be skipped while write to file
  2. INSERT INTO TABLE - complex type but different names
  3. Insert overwrite table command should output correct schema: basic
  4. parquet timestamp conversion
  5. SPARK-29174 Support LOCAL in INSERT OVERWRITE DIRECTORY to data source
  6. SPARK-33901: ctas should should not change table's schema
  7. SPARK-37160: CREATE TABLE AS SELECT with CHAR_AS_VARCHAR
  8. SPARK-38336 INSERT INTO statements with tables with default columns: positive tests
  9. SPARK-38811 INSERT INTO on columns added with ALTER TABLE ADD COLUMNS: Positive tests
  10. SPARK-43071: INSERT INTO from queries whose final operators are not projections
  11. write path implements onTaskCommit API correctly
  12. Write Spark version into Parquet metadata

Additional tests which had to be ignored for spark 4 : 

  1. ctas with union
  2. SPARK-48817: test multi inserts

  Total: 12 tests (Spark 3.4/3.5) | 14 tests (Spark 4.0)