Skip to content

[SPARK-55128][INFRA][FOLLOWUP] Restore SQL tests by pin 'pandas==2.3.3' for maven daily test#53933

Closed
LuciferYang wants to merge 1 commit intoapache:masterfrom
LuciferYang:SPARK-55128-FOLLOWUP
Closed

[SPARK-55128][INFRA][FOLLOWUP] Restore SQL tests by pin 'pandas==2.3.3' for maven daily test#53933
LuciferYang wants to merge 1 commit intoapache:masterfrom
LuciferYang:SPARK-55128-FOLLOWUP

Conversation

@LuciferYang
Copy link
Contributor

What changes were proposed in this pull request?

Similar to #53910, this pr pins the pandas version to 2.3.3.

Why are the changes needed?

To restore SQL tests for maven daily test.

- udf/postgreSQL/udf-case.sql - Scalar Pandas UDF *** FAILED ***
  udf/postgreSQL/udf-case.sql - Scalar Pandas UDF
  Python: 3.11 Pandas: 3.0.0 PyArrow: 23.0.0
  Expected Some("struct<Two:string,i:int,f:double,i:int,j:int>"), but got Some("struct<>") Schema did not match for query #30
  SELECT '' AS `Two`, *
    FROM CASE_TBL a, CASE2_TBL b
    WHERE udf(COALESCE(f,b.i) = 2): -- !query
  SELECT '' AS `Two`, *
    FROM CASE_TBL a, CASE2_TBL b
    WHERE udf(COALESCE(f,b.i) = 2)
  -- !query schema
  struct<>
  -- !query output
  org.apache.spark.SparkRuntimeException
  {
    "errorClass" : "CAST_INVALID_INPUT",
    "sqlState" : "22018",
    "messageParameters" : {
      "ansiConfig" : "\"spark.sql.ansi.enabled\"",
      "expression" : "'nan'",
      "sourceType" : "\"STRING\"",
      "targetType" : "\"BOOLEAN\""
    },
    "queryContext" : [ {
      "objectType" : "",
      "objectName" : "",
      "startIndex" : 62,
      "stopIndex" : 85,
      "fragment" : "udf(COALESCE(f,b.i) = 2)"
    } ]
  } (SQLQueryTestSuite.scala:681)

Does this PR introduce any user-facing change?

No

How was this patch tested?

monitor maven daily test after pr merged

Was this patch authored or co-authored using generative AI tooling?

No

@github-actions
Copy link

JIRA Issue Information

=== Test SPARK-55128 ===
Summary: Restore SQL tests by pin 'pandas<3'
Assignee: None
Status: Resolved
Affected: ["4.2"]


This comment was automatically generated by GitHub Actions

@github-actions github-actions bot added the INFRA label Jan 23, 2026
@zhengruifeng
Copy link
Contributor

thanks, merged to master

@LuciferYang
Copy link
Contributor Author

Thank you @zhengruifeng and @HyukjinKwon

LuciferYang added a commit that referenced this pull request Jan 23, 2026
…3' for maven daily test

### What changes were proposed in this pull request?
Similar to #53910, this pr pins the pandas version to 2.3.3.

### Why are the changes needed?
To  restore SQL tests for maven daily test.
- https://github.com/apache/spark/actions/runs/21249870076/job/61148348328

```
- udf/postgreSQL/udf-case.sql - Scalar Pandas UDF *** FAILED ***
  udf/postgreSQL/udf-case.sql - Scalar Pandas UDF
  Python: 3.11 Pandas: 3.0.0 PyArrow: 23.0.0
  Expected Some("struct<Two:string,i:int,f:double,i:int,j:int>"), but got Some("struct<>") Schema did not match for query #30
  SELECT '' AS `Two`, *
    FROM CASE_TBL a, CASE2_TBL b
    WHERE udf(COALESCE(f,b.i) = 2): -- !query
  SELECT '' AS `Two`, *
    FROM CASE_TBL a, CASE2_TBL b
    WHERE udf(COALESCE(f,b.i) = 2)
  -- !query schema
  struct<>
  -- !query output
  org.apache.spark.SparkRuntimeException
  {
    "errorClass" : "CAST_INVALID_INPUT",
    "sqlState" : "22018",
    "messageParameters" : {
      "ansiConfig" : "\"spark.sql.ansi.enabled\"",
      "expression" : "'nan'",
      "sourceType" : "\"STRING\"",
      "targetType" : "\"BOOLEAN\""
    },
    "queryContext" : [ {
      "objectType" : "",
      "objectName" : "",
      "startIndex" : 62,
      "stopIndex" : 85,
      "fragment" : "udf(COALESCE(f,b.i) = 2)"
    } ]
  } (SQLQueryTestSuite.scala:681)
```

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
monitor maven daily test after pr merged

### Was this patch authored or co-authored using generative AI tooling?
No

Closes #53933 from LuciferYang/SPARK-55128-FOLLOWUP.

Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>
(cherry picked from commit 3f1c9a3)
Signed-off-by: yangjie01 <yangjie01@baidu.com>
@LuciferYang
Copy link
Contributor Author

thanks, merged to master

Merged into branch-4.1 too.

pan3793 pushed a commit to pan3793/spark that referenced this pull request Feb 11, 2026
…3' for maven daily test

Similar to apache#53910, this pr pins the pandas version to 2.3.3.

To  restore SQL tests for maven daily test.
- https://github.com/apache/spark/actions/runs/21249870076/job/61148348328

```
- udf/postgreSQL/udf-case.sql - Scalar Pandas UDF *** FAILED ***
  udf/postgreSQL/udf-case.sql - Scalar Pandas UDF
  Python: 3.11 Pandas: 3.0.0 PyArrow: 23.0.0
  Expected Some("struct<Two:string,i:int,f:double,i:int,j:int>"), but got Some("struct<>") Schema did not match for query apache#30
  SELECT '' AS `Two`, *
    FROM CASE_TBL a, CASE2_TBL b
    WHERE udf(COALESCE(f,b.i) = 2): -- !query
  SELECT '' AS `Two`, *
    FROM CASE_TBL a, CASE2_TBL b
    WHERE udf(COALESCE(f,b.i) = 2)
  -- !query schema
  struct<>
  -- !query output
  org.apache.spark.SparkRuntimeException
  {
    "errorClass" : "CAST_INVALID_INPUT",
    "sqlState" : "22018",
    "messageParameters" : {
      "ansiConfig" : "\"spark.sql.ansi.enabled\"",
      "expression" : "'nan'",
      "sourceType" : "\"STRING\"",
      "targetType" : "\"BOOLEAN\""
    },
    "queryContext" : [ {
      "objectType" : "",
      "objectName" : "",
      "startIndex" : 62,
      "stopIndex" : 85,
      "fragment" : "udf(COALESCE(f,b.i) = 2)"
    } ]
  } (SQLQueryTestSuite.scala:681)
```

No

monitor maven daily test after pr merged

No

Closes apache#53933 from LuciferYang/SPARK-55128-FOLLOWUP.

Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>
pan3793 pushed a commit that referenced this pull request Feb 11, 2026
Backport #53910 and #53933 to branch-4.0

### What changes were proposed in this pull request?

Pandas 3.0 released, pin 'pandas==2.3.3' to recover the CI.

### Why are the changes needed?

Recover CI.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Wait for GHA result.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #54263 from pan3793/SPARK-55128-4.0.

Lead-authored-by: Ruifeng Zheng <ruifengz@apache.org>
Co-authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants