[SPARK-40846][INFRA] Temporarily pin GA used Java version to pass GA first #38311

LuciferYang · 2022-10-19T11:40:56Z

What changes were proposed in this pull request?

This pr aims to pin GA used Java version to pass GA test first:

Java 8 pin to 8.0.345
Java 11 pin to 11.0.16
Java 17 pin to 17.0.4

this change should be revert after find the root cause

Why are the changes needed?

Make GA passed first.

The following test failed with 8u352/11.0.17/17.0.5:

[info] *** 12 TESTS FAILED ***
[error] Failed: Total 6746, Failed 12, Errors 0, Passed 6734, Ignored 5
[error] Failed tests:
[error] 	org.apache.spark.sql.catalyst.expressions.CastWithAnsiOffSuite
[error] 	org.apache.spark.sql.catalyst.util.TimestampFormatterSuite
[error] 	org.apache.spark.sql.catalyst.expressions.CastWithAnsiOnSuite
[error] 	org.apache.spark.sql.catalyst.util.RebaseDateTimeSuite
[error] 	org.apache.spark.sql.catalyst.expressions.TryCastSuite

Does this PR introduce any user-facing change?

No

How was this patch tested?

Pass GitHub Actions

LuciferYang · 2022-10-19T11:44:19Z

.github/workflows/build_and_test.yml

        required: false
        type: string
-        default: 8
+        default: 8.0.345


@HyukjinKwon @Yikun Can the version be written like this

so the actual version was changed from time to time?

Yes, always use the latest version

got it, thanks

https://github.com/actions/setup-java/blob/main/src/distributions/zulu/installer.ts#L30-L45

Yes, we are using setup-java (zulu), and it find the latest java.

As we offline discussed, we could first to pin java8/java11/java17 to specific version to recover CI.

LuciferYang · 2022-10-19T12:01:50Z

Failed links:

LuciferYang · 2022-10-19T12:11:35Z

I can reproduce the failed with Java 8u352, for example

run

build/sbt clean "catalyst/testOnly *CastWithAnsiOffSuite"

with 8u352

[info] - SPARK-35711: cast timestamp without time zone to timestamp with local time zone *** FAILED *** (190 milliseconds)
[info]   Incorrect evaluation (codegen off): cast(0001-01-01 00:00:00 as timestamp), actual: -62135617820000000, expected: -62135596800000000 (ExpressionEvalHelper.scala:209)
[info]   org.scalatest.exceptions.TestFailedException:
[info]   at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
[info]   at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
[info]   at org.scalatest.funsuite.AnyFunSuite.newAssertionFailedException(AnyFunSuite.scala:1564)
[info]   at org.scalatest.Assertions.fail(Assertions.scala:933)
[info]   at org.scalatest.Assertions.fail$(Assertions.scala:929)
[info]   at org.scalatest.funsuite.AnyFunSuite.fail(AnyFunSuite.scala:1564)
[info]   at org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper.checkEvaluationWithoutCodegen(ExpressionEvalHelper.scala:209)
[info]   at org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper.checkEvaluationWithoutCodegen$(ExpressionEvalHelper.scala:199)
[info]   at org.apache.spark.sql.catalyst.expressions.CastSuiteBase.checkEvaluationWithoutCodegen(CastSuiteBase.scala:49)
[info]   at org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper.checkEvaluation(ExpressionEvalHelper.scala:87)
[info]   at org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper.checkEvaluation$(ExpressionEvalHelper.scala:82)
[info]   at org.apache.spark.sql.catalyst.expressions.CastSuiteBase.checkEvaluation(CastSuiteBase.scala:49)
[info]   at org.apache.spark.sql.catalyst.expressions.CastSuiteBase.$anonfun$new$198(CastSuiteBase.scala:893)
[info]   at org.apache.spark.sql.catalyst.expressions.CastSuiteBase.$anonfun$new$198$adapted(CastSuiteBase.scala:890)
[info]   at scala.collection.immutable.List.foreach(List.scala:431)
[info]   at org.apache.spark.sql.catalyst.expressions.CastSuiteBase.$anonfun$new$197(CastSuiteBase.scala:890)
[info]   at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[info]   at org.apache.spark.sql.catalyst.util.DateTimeTestUtils$.withDefaultTimeZone(DateTimeTestUtils.scala:61)
[info]   at org.apache.spark.sql.catalyst.expressions.CastSuiteBase.$anonfun$new$196(CastSuiteBase.scala:890)
[info]   at org.apache.spark.sql.catalyst.expressions.CastSuiteBase.$anonfun$new$196$adapted(CastSuiteBase.scala:888)
[info]   at scala.collection.immutable.List.foreach(List.scala:431)
[info]   at org.apache.spark.sql.catalyst.expressions.CastSuiteBase.$anonfun$new$195(CastSuiteBase.scala:888)
[info]   at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:20)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
[info]   at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:207)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
[info]   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
[info]   at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:66)
[info]   at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
[info]   at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
[info]   at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:66)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
[info]   at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
[info]   at scala.collection.immutable.List.foreach(List.scala:431)
[info]   at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
[info]   at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
[info]   at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
[info]   at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
[info]   at org.scalatest.Suite.run(Suite.scala:1114)
[info]   at org.scalatest.Suite.run$(Suite.scala:1096)
[info]   at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
[info]   at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
[info]   at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:66)
[info]   at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
[info]   at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info]   at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info]   at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:66)
[info]   at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info]   at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:413)
[info]   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[info]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[info]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[info]   at java.lang.Thread.run(Thread.java:750)

and run them with 8u345:

[info] Run completed in 49 seconds, 54 milliseconds.
[info] Total number of tests run: 80
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 80, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.

LuciferYang · 2022-10-19T12:12:26Z

also ping @dongjoon-hyun @srowen @wangyum

srowen · 2022-10-19T12:13:36Z

It feels like something we (also) need to fix; I assume users will use later Java 8 versions, too. CC @MaxGekk

LuciferYang · 2022-10-19T12:15:44Z

Yes, the 8u345 can pass, but the 8u352 cannot, need to investigate. Do we need to restore the GA first?

LuciferYang · 2022-10-19T12:21:22Z

Let me manually test the latest Java 11 and 17

srowen · 2022-10-19T12:30:18Z

Right, I think it's OK to restore GA, but, users will hit this

LuciferYang · 2022-10-19T12:31:01Z

hmm... the latest 11(11.0.17) and 17(17.0.5) have the same issue ...

srowen · 2022-10-19T12:39:17Z

This isn't the same as https://issues.apache.org/jira/browse/SPARK-40791 is it?

MaxGekk

hmm... the latest 11(11.0.17) and 17(17.0.5) have the same issue ...

The latest JDKs can have more precise info about time zone offsets. Let's restore GAs and open a separate JIRA to investigate the failures and adjust our tests.

LuciferYang · 2022-10-19T12:43:15Z

@MaxGekk should we need to ping the version of Java 11 and 17 to the previous version? The daily test with Java 11 and Java 17 will also fail,

MaxGekk · 2022-10-19T12:46:53Z

should we need to ping the version of Java 11 and 17 to the previous version?

Yes, I think it makes sense.

LuciferYang · 2022-10-19T16:47:28Z

GA passed

HyukjinKwon · 2022-10-20T00:15:37Z

Okay, I am merging this but would you mind filing a blocker JIRA for this issue @LuciferYang ?

HyukjinKwon · 2022-10-20T00:15:55Z

We should make sure this is a temporary fix.

HyukjinKwon · 2022-10-20T00:16:07Z

Merged to master.

LuciferYang · 2022-10-20T03:34:15Z

Okay, I am merging this but would you mind filing a blocker JIRA for this issue @LuciferYang ?

https://issues.apache.org/jira/browse/SPARK-40851

wangyum · 2022-10-20T04:20:18Z

This is because Antarctica/Vostok has been removed. Please see: openjdk/jdk8u-dev@46da2c3#diff-e1d56ed7eaec2a5fa47dbca1d570491df64877e9efcb52065af7f250709ce0e4

I think we can also remove Antarctica/Vostok from our tests:

./catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeTestUtils.scala:    "Antarctica/Vostok",
./catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/TimestampFormatterSuite.scala:      "Antarctica/Vostok" -> 1543723872001234L,
./catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/TimestampFormatterSuite.scala:      "Antarctica/Vostok" -> "2018-12-02 16:11:12.001234",
./catalyst/src/main/resources/julian-gregorian-rebase-micros.json:  "tz" : "Antarctica/Vostok",
./catalyst/src/main/resources/gregorian-julian-rebase-micros.json:  "tz" : "Antarctica/Vostok",

MaxGekk · 2022-10-20T05:28:15Z

Highly likely, we will need to regenerate the JSON files by the tests

spark/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/RebaseDateTimeSuite.scala

Lines 309 to 323 in 3faced8

    
           ignore("generate 'gregorian-julian-rebase-micros.json'") { 
        
             generateRebaseJson( 
        
               adjustFunc = (_: TimeZone, micros: Long) => micros, 
        
               rebaseFunc = rebaseGregorianToJulianMicros, 
        
               dir = "/Users/maximgekk/tmp", 
        
               fileName = "gregorian-julian-rebase-micros.json") 
        
           } 
        
           ignore("generate 'julian-gregorian-rebase-micros.json'") { 
        
             generateRebaseJson( 
        
               adjustFunc = rebaseGregorianToJulianMicros, 
        
               rebaseFunc = rebaseJulianToGregorianMicros, 
        
               dir = "/Users/maximgekk/tmp", 
        
               fileName = "julian-gregorian-rebase-micros.json") 
        
           }

./catalyst/src/main/resources/julian-gregorian-rebase-micros.json:  "tz" : "Antarctica/Vostok",
./catalyst/src/main/resources/gregorian-julian-rebase-micros.json:  "tz" : "Antarctica/Vostok",

LuciferYang · 2022-10-20T06:05:27Z

Highly likely, we will need to regenerate the JSON files by the tests

spark/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/RebaseDateTimeSuite.scala

Lines 309 to 323 in 3faced8

ignore("generate 'gregorian-julian-rebase-micros.json'") {

generateRebaseJson(

adjustFunc = (_: TimeZone, micros: Long) => micros,

rebaseFunc = rebaseGregorianToJulianMicros,

dir = "/Users/maximgekk/tmp",

fileName = "gregorian-julian-rebase-micros.json")

}

ignore("generate 'julian-gregorian-rebase-micros.json'") {

generateRebaseJson(

adjustFunc = rebaseGregorianToJulianMicros,

rebaseFunc = rebaseJulianToGregorianMicros,

dir = "/Users/maximgekk/tmp",

fileName = "julian-gregorian-rebase-micros.json")

}
./catalyst/src/main/resources/julian-gregorian-rebase-micros.json:  "tz" : "Antarctica/Vostok",
./catalyst/src/main/resources/gregorian-julian-rebase-micros.json:  "tz" : "Antarctica/Vostok",

OK

LuciferYang · 2022-10-20T08:43:47Z

This isn't the same as https://issues.apache.org/jira/browse/SPARK-40791 is it?

@srowen They are different. SPARK-40791 is cause by change of Java 19

…first ### What changes were proposed in this pull request? This pr aims to pin GA used Java version to pass GA test first: - Java 8 pin to 8.0.345 - Java 11 pin to 11.0.16 - Java 17 pin to 17.0.4 this change should be revert after find the root cause ### Why are the changes needed? Make GA passed first. The following test failed with 8u352/11.0.17/17.0.5: ``` [info] *** 12 TESTS FAILED *** [error] Failed: Total 6746, Failed 12, Errors 0, Passed 6734, Ignored 5 [error] Failed tests: [error] org.apache.spark.sql.catalyst.expressions.CastWithAnsiOffSuite [error] org.apache.spark.sql.catalyst.util.TimestampFormatterSuite [error] org.apache.spark.sql.catalyst.expressions.CastWithAnsiOnSuite [error] org.apache.spark.sql.catalyst.util.RebaseDateTimeSuite [error] org.apache.spark.sql.catalyst.expressions.TryCastSuite ``` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Pass GitHub Actions Closes apache#38311 from LuciferYang/java-version. Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

pin 345

a41e9e1

github-actions bot added the INFRA label Oct 19, 2022

LuciferYang changed the title ~~[DON'T MERGE]~~ [DON'T MERGE] Try pin Java 8 to 8.0.345 Oct 19, 2022

LuciferYang changed the title ~~[DON'T MERGE] Try pin Java 8 to 8.0.345~~ [DON'T MERGE][INFRA] Try pin Java 8 to 8.0.345 Oct 19, 2022

LuciferYang commented Oct 19, 2022

View reviewed changes

LuciferYang changed the title ~~[DON'T MERGE][INFRA] Try pin Java 8 to 8.0.345~~ [DON'T MERGE][INFRA] Try pin Java 8 version to 8.0.345 Oct 19, 2022

LuciferYang changed the title ~~[DON'T MERGE][INFRA] Try pin Java 8 version to 8.0.345~~ [SPARK-40846][INFRA] Try pin Java 8 version to 8.0.345 Oct 19, 2022

LuciferYang changed the title ~~[SPARK-40846][INFRA] Try pin Java 8 version to 8.0.345~~ [SPARK-40846][INFRA] Pin default Java 8 version to 8.0.345 Oct 19, 2022

LuciferYang changed the title ~~[SPARK-40846][INFRA] Pin default Java 8 version to 8.0.345~~ [SPARK-40846][INFRA] Pin GA used Java 8 default version to 8.0.345 Oct 19, 2022

LuciferYang changed the title ~~[SPARK-40846][INFRA] Pin GA used Java 8 default version to 8.0.345~~ [SPARK-40846][INFRA] Temporarily pin GA used Java 8 default version to 8.0.345 Oct 19, 2022

MaxGekk reviewed Oct 19, 2022

View reviewed changes

pin Java 11 and Java 17

de28549

LuciferYang changed the title ~~[SPARK-40846][INFRA] Temporarily pin GA used Java 8 default version to 8.0.345~~ [SPARK-40846][INFRA] Temporarily pin GA used Java version to pass GA first Oct 19, 2022

revert maven build

fcf09fc

HyukjinKwon approved these changes Oct 20, 2022

View reviewed changes

HyukjinKwon closed this in 1311976 Oct 20, 2022

LuciferYang mentioned this pull request Oct 21, 2022

[SPARK-40800][SQL] Always inline expressions in OptimizeOneRowRelationSubquery #38260

Closed

[SPARK-40846][INFRA] Temporarily pin GA used Java version to pass GA first #38311

[SPARK-40846][INFRA] Temporarily pin GA used Java version to pass GA first #38311

Uh oh!

Conversation

LuciferYang commented Oct 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

LuciferYang Oct 19, 2022

Choose a reason for hiding this comment

Uh oh!

zhengruifeng Oct 19, 2022

Choose a reason for hiding this comment

Uh oh!

LuciferYang Oct 19, 2022

Choose a reason for hiding this comment

Uh oh!

zhengruifeng Oct 19, 2022

Choose a reason for hiding this comment

Uh oh!

Yikun Oct 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LuciferYang commented Oct 19, 2022

Uh oh!

LuciferYang commented Oct 19, 2022

Uh oh!

LuciferYang commented Oct 19, 2022

Uh oh!

srowen commented Oct 19, 2022

Uh oh!

LuciferYang commented Oct 19, 2022

Uh oh!

LuciferYang commented Oct 19, 2022

Uh oh!

srowen commented Oct 19, 2022

Uh oh!

LuciferYang commented Oct 19, 2022

Uh oh!

srowen commented Oct 19, 2022

Uh oh!

MaxGekk left a comment

Choose a reason for hiding this comment

Uh oh!

LuciferYang commented Oct 19, 2022

Uh oh!

MaxGekk commented Oct 19, 2022

Uh oh!

LuciferYang commented Oct 19, 2022

Uh oh!

HyukjinKwon commented Oct 20, 2022

Uh oh!

HyukjinKwon commented Oct 20, 2022

Uh oh!

HyukjinKwon commented Oct 20, 2022

Uh oh!

LuciferYang commented Oct 20, 2022

Uh oh!

wangyum commented Oct 20, 2022

Uh oh!

MaxGekk commented Oct 20, 2022

Uh oh!

LuciferYang commented Oct 20, 2022

Uh oh!

LuciferYang commented Oct 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

LuciferYang commented Oct 19, 2022 •

edited

Loading

Yikun Oct 19, 2022 •

edited

Loading

LuciferYang commented Oct 20, 2022 •

edited

Loading