[GLUTEN-11355][UT] Add new Spark 4.1 tests by baibaichen · Pull Request #11380 · apache/incubator-gluten

baibaichen · 2026-01-08T09:44:10Z

This PR automates the migration of Gluten test and SQL Test Suites from spark 4.1
Fixes #11355

How to Generate Spark 4.1 Gluten Test Suites

With the help of AI, we follow a four-step process—details can be found here

1 Baseline Analysis

Scan existing Gluten Spark 4.0 test suites (Gluten*Suite.scala)
Map each Gluten suite to its corresponding Spark 4.0 source file
Generate mapping report with found/not-found status

2 Package Scope Extraction

Extract unique package paths from successfully mapped suites
Create a focused list of directories to monitor for changes

3 Delta Detection

Compare Spark 4.0 vs Spark 4.1 test files in identified packages
Identify net-new *Suite.scala files introduced in Spark 4.1

4 Code Generation

For each new Spark 4.1 suite, generate corresponding Gluten wrapper
Preserve original package structure
Auto-create directory hierarchy as needed

Result: Automated generation of 27 Gluten test suites for gluten Spark 4.1, ensuring test coverage parity with upstream changes.

How to Migrate SQL Test Suites to Spark 4.1

Three-Way Git Merge Approach

This PR migrates SQL test suites (inputs/*. sql and results/*.sql.out) from Spark 4.1 using Git's three-way merge algorithm to preserve both upstream changes and Gluten-specific modifications.

Result:

Auto-merged: 165 files
New tests added: 31 files (collations, edge cases, recursion, spatial, etc.)
Modified tests: 134 files
Deleted tests: 2 files (collations.sql -> split into 4 files, timestamp-ntz.sql)

Fix and Exclusion

Cause	Type	Category	Description	Affected files
#52406	4.1.0	Test Exclusion	Exclude "infer shredding with mixed scale" test. Shredding behavior with mixed decimal scales changed in Spark 4.1.	gluten-ut/spark41/.../velox/VeloxTestSettings.scala Excluded test: - "infer shredding with mixed scale" in GlutenFileBasedDataSourceSuite
#50599	Fix	Bug Fix	Implement Kryo serialization for CachedColumnarBatch. Adds proper Kryo serialization support for columnar cache to work with Spark's Kryo-based serialization.	backends-velox/.../ColumnarCachedBatchSerializer.scala gluten-ut/spark41/.../GlutenCacheTableInKryoSuite.scala
#50230	4.1.0	Test Exclusion	Exclude GlutenMapStatusEndToEndSuite and configure parallelism. MapStatus serialization behavior changed in Spark 4.1 requiring test adjustments.	gluten-ut/spark41/.../velox/VeloxTestSettings.scala gluten-ut/spark41/.../GlutenMapStatusEndToEndSuite.scala Excluded tests: - GlutenMapStatusEndToEndSuite (entire suite)
#52473 #52870 #52891	4.1.0	Test Exclusion	Exclude Spark Structured Streaming tests in Gluten. Multiple streaming-related tests need fixes due to upstream Spark changes.	gluten-ut/spark41/.../velox/VeloxTestSettings.scala Excluded tests: - GlutenStreamRealTimeModeAllowlistSuite: "rtm operator allowlist", "repartition not allowed", "stateful queries not allowed" - GlutenStreamRealTimeModeE2ESuite: "foreach", "to_json and from_json round-trip", "generateExec passthrough" - GlutenStreamRealTimeModeSuite: "processAllAvailable"
-	4.1.0	Test Exclusion	Exclude failing SQL tests on Spark 4.1. Multiple SQL test files need to be fixed for Spark 4.1 compatibility.	gluten-ut/spark41/.../velox/VeloxSQLQueryTestSettings.scala Excluded tests: - cast.sql - describe.sql - nonansi/cast.sql - nonansi/st-functions.sql - scripting/randomly_generated_scripts.sql - st-functions.sql - type-coercion-edge-cases.sql - variant-field-extractions.sql
#50287	Fix	Bug Fix	Replace `RuntimeReplaceable` with its `replacement` to fix unit tests. This handles the RuntimeReplaceable expression changes in Spark to ensure proper expression transformation.	gluten-ut/common/.../GlutenTestsTrait.scala gluten-ut/spark41/.../GlutenTimeExpressionsSuite.scala gluten-ut/spark41/.../shim/GlutenTestsTrait.scala

How was this patch tested?

Run new tests.

github-actions · 2026-01-08T09:54:32Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-01-08T10:23:01Z

Run Gluten Clickhouse CI on x86

baibaichen · 2026-01-08T10:32:27Z

.github/workflows/clickhouse_be_trigger.yml

+      - 'shims/spark40/**'
+      - 'shims/spark41/**'
+      - 'gluten-ut/spark40/**'
+      - 'gluten-ut/spark41/**'


@zzcclp @lwz9103
Since the ClickHouse backend does not support Spark 4.0 and later, I ignored it. I also ignored clickhouse_be_trigger.yml. This should be fine.

It's OK to me.

github-actions · 2026-01-09T09:19:29Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-01-12T10:00:18Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-01-12T14:56:19Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-01-13T02:43:56Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-01-13T08:17:55Z

Run Gluten Clickhouse CI on x86

…e#11120

…ctories for clickhouse backend

Three-way merge performed using Git: - Base: Spark 4.0.1 (29434ea766b) - Left: Spark 4.1.0 (e221b56be7b) - Right: Gluten Spark 4.1 backends-velox Summary: - Auto-merged: 165 files - New tests added: 31 files (collations, edge cases, recursion, spatial, etc.) - Modified tests: 134 files - Deleted tests: 2 files (collations.sql -> split into 4 files, timestamp-ntz.sql) Conflicts resolved: - inputs/timestamp-ntz.sql: Right deleted + Left modified -> DELETED (per resolution rule) New test suites from Spark 4.1.0: - Collations (4 files): aliases, basic, padding-trim, string-functions - Edge cases (6 files): alias-resolution, extract-value, join-resolution, etc. - Advanced features: cte-recursion, generators, kllquantiles, thetasketch, time - Name resolution: order-by-alias, session-variable-precedence, runtime-replaceable - Spatial functions: st-functions (ANSI and non-ANSI variants) - Various resolution edge cases Total files after merge: 671 (up from 613)

see apache/spark#50287

see apache/spark#52406

see apache/spark#50599

see apache/spark#50230

see - apache/spark#52473 - apache/spark#52870 - apache/spark#52891

…dBatchSerializer to work across different spark versions

see apache/spark#50287

…efix

…eTrait

github-actions · 2026-01-13T12:32:07Z

Run Gluten Clickhouse CI on x86

zhouyuan

👍

github-actions bot added the CORE works for Gluten Core label Jan 8, 2026

github-actions bot added the INFRA label Jan 8, 2026

baibaichen commented Jan 8, 2026

View reviewed changes

baibaichen force-pushed the feature/41_ut_ext branch from 319fe1b to 97dc360 Compare January 9, 2026 09:18

github-actions bot added BUILD TOOLS labels Jan 9, 2026

baibaichen force-pushed the feature/41_ut_ext branch from 97dc360 to 4b2224c Compare January 12, 2026 09:59

github-actions bot added the VELOX label Jan 12, 2026

baibaichen marked this pull request as ready for review January 12, 2026 10:03

baibaichen force-pushed the feature/41_ut_ext branch from ffeb840 to a291da4 Compare January 13, 2026 02:43

baibaichen mentioned this pull request Jan 13, 2026

[GLUTEN-11027][VL][CI] Add clang-tidy to check cpp code #11120

Merged

[Fix] Update artifact download path for velox-native-lib due to apach…

cea0f39

…e#11120

baibaichen mentioned this pull request Jan 13, 2026

[VL] Track on Spark-4.1.x failed unit tests #11400

Open

baibaichen and others added 10 commits January 13, 2026 12:12

Move GlutenStreamingQuerySuite to correct package

c0b7848

Add Spark 4.1 new test suites for Gluten

26b2a5f

Enable new and existing Gluten test suites for Spark 4.1 UT

19aab2e

Update workflow trigger paths to exclude Spark 4.0 and 4.1 shims dire…

5a8034c

…ctories for clickhouse backend

Add support for Spark 4.1 in build script

e7fb255

Enable additional Spark 4.1 SQL tests by resolving TODOs

947cd4c

Add new Spark 4.1 test files to VeloxSQLQueryTestSettings

6b893da

[Fix] Replace RuntimeReplaceable with its replacement to fix UT.

e77762e

see apache/spark#50287

[4.1.0] Exclude "infer shredding with mixed scale"

f8d0963

see apache/spark#52406

baibaichen and others added 8 commits January 13, 2026 12:12

[Fix] Implement Kryo serialization for CachedColumnarBatch

5cc41c6

see apache/spark#50599

[4.1.0] Exclude GlutenMapStatusEndToEndSuite and configure parallelism

4eee40e

see apache/spark#50230

[4.1.0] Exclude Spark Structure Steaming tests in Gluten

31bc8f4

see - apache/spark#52473 - apache/spark#52870 - apache/spark#52891

[4.1.0] Exclude failing SQL tests on Spark 4.1

ec5c0fb

Replace SparkException.require with standard require in ColumnarCache…

8944bb2

…dBatchSerializer to work across different spark versions

[Fix] Replace RuntimeReplaceable with its replacement to fix UT.

80461d5

see apache/spark#50287

Exclude Spark 4.0 and 4.1 paths in clickhouse_be_trigger using ! pr…

fa54494

…efix

[Fix] Update GlutenShowNamespacesParserSuite to use GlutenSQLTestsBas…

6526606

…eTrait

baibaichen force-pushed the feature/41_ut_ext branch from 545c5a1 to 6526606 Compare January 13, 2026 12:31

zhouyuan approved these changes Jan 13, 2026

View reviewed changes

baibaichen merged commit f194cbe into apache:main Jan 13, 2026
117 of 119 checks passed

baibaichen deleted the feature/41_ut_ext branch January 13, 2026 23:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[GLUTEN-11355][UT] Add new Spark 4.1 tests#11380

[GLUTEN-11355][UT] Add new Spark 4.1 tests#11380
baibaichen merged 19 commits intoapache:mainfrom
baibaichen:feature/41_ut_ext

baibaichen commented Jan 8, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Jan 8, 2026

Uh oh!

github-actions bot commented Jan 8, 2026

Uh oh!

baibaichen Jan 8, 2026

Uh oh!

zzcclp Jan 8, 2026

Uh oh!

lwz9103 Jan 8, 2026

Uh oh!

github-actions bot commented Jan 9, 2026

Uh oh!

github-actions bot commented Jan 12, 2026

Uh oh!

github-actions bot commented Jan 12, 2026

Uh oh!

github-actions bot commented Jan 13, 2026

Uh oh!

github-actions bot commented Jan 13, 2026

Uh oh!

github-actions bot commented Jan 13, 2026

Uh oh!

zhouyuan left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

baibaichen commented Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How to Generate Spark 4.1 Gluten Test Suites

1 Baseline Analysis

2 Package Scope Extraction

3 Delta Detection

4 Code Generation

How to Migrate SQL Test Suites to Spark 4.1

Three-Way Git Merge Approach

Fix and Exclusion

How was this patch tested?

Uh oh!

github-actions bot commented Jan 8, 2026

Uh oh!

github-actions bot commented Jan 8, 2026

Uh oh!

baibaichen Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

zzcclp Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

lwz9103 Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jan 9, 2026

Uh oh!

github-actions bot commented Jan 12, 2026

Uh oh!

github-actions bot commented Jan 12, 2026

Uh oh!

github-actions bot commented Jan 13, 2026

Uh oh!

github-actions bot commented Jan 13, 2026

Uh oh!

github-actions bot commented Jan 13, 2026

Uh oh!

zhouyuan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

baibaichen commented Jan 8, 2026 •

edited

Loading