Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ZEPPELIN-6090] Drop JDK8 from CI #4875

Merged
merged 1 commit into from
Oct 22, 2024
Merged

Conversation

Reamer
Copy link
Contributor

@Reamer Reamer commented Oct 17, 2024

What is this PR for?

This pull request removes the test with JDK 8. I have kept the JDK matrix to make it easier to switch to the next JDK later.
This pull request also drops python 3.7 & 3.8 tests for the python,rlang and zeppelin-jupyter interpreter. Please note that current python 3.7 and 3.8 are end of life. https://devguide.python.org/versions/

What type of PR is it?

Improvement

What is the Jira issue?

How should this be tested?

  • CI

Questions:

  • Does the license files need to update? No
  • Is there breaking changes for older versions? No
  • Does this needs documentation? No

@Reamer Reamer changed the title Jdk11 GitHub [ZEPPELIN-6090] Drop JDK8 from CI Oct 17, 2024
jongyoul
jongyoul previously approved these changes Oct 17, 2024
Copy link
Member

@jongyoul jongyoul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Let's wait for CI

run: |
./mvnw install -DskipTests -pl livy -am ${MAVEN_ARGS}
./testing/downloadSpark.sh "2.4.8" "2.7"
./testing/downloadLivy.sh "0.8.0-incubating" "2.11"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Livy 0.8 also provides 2.12 binary, which should work with Spark 3 and may support Java 11, though I haven't tested.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll give it a try.Reamer@e6f0067

@pan3793
Copy link
Member

pan3793 commented Oct 18, 2024

@Reamer Livy 0.8 seems tested with Spark 3.2.3
https://github.com/apache/incubator-livy/blob/v0.8.0-incubating/dev/docker/livy-dev-spark/Dockerfile#L20

@Reamer
Copy link
Contributor Author

Reamer commented Oct 18, 2024

I am currently testing with Spark version 3.5 and the tests are running except for one.

Error:  Failures: 
Error:    LivyInterpreterIT.testPySparkInterpreter:407 %text 'JavaPackage' object is not callable
Traceback (most recent call last):
  File "/home/runner/work/zeppelin/zeppelin/spark-3.5.3-bin-hadoop3/python/lib/pyspark.zip/pyspark/sql/session.py", line 1444, in createDataFrame
    data, schema, samplingRatio, verifySchema  # type: ignore[arg-type]
  File "/home/runner/work/zeppelin/zeppelin/spark-3.5.3-bin-hadoop3/python/lib/pyspark.zip/pyspark/sql/session.py", line 1485, in _create_dataframe
    rdd, struct = self._createFromLocal(map(prepare, data), schema)
  File "/home/runner/work/zeppelin/zeppelin/spark-3.5.3-bin-hadoop3/python/lib/pyspark.zip/pyspark/sql/session.py", line 1093, in _createFromLocal
    struct = self._inferSchemaFromList(data, names=schema)
  File "/home/runner/work/zeppelin/zeppelin/spark-3.5.3-bin-hadoop3/python/lib/pyspark.zip/pyspark/sql/session.py", line 954, in _inferSchemaFromList
    prefer_timestamp_ntz = is_timestamp_ntz_preferred()
  File "/home/runner/work/zeppelin/zeppelin/spark-3.5.3-bin-hadoop3/python/lib/pyspark.zip/pyspark/sql/utils.py", line 153, in is_timestamp_ntz_preferred
    return jvm is not None and jvm.PythonSQLUtils.isTimestampNTZPreferred()
TypeError: 'JavaPackage' object is not callable
 ==> expected: <SUCCESS> but was: <ERROR>

@pan3793
Copy link
Member

pan3793 commented Oct 18, 2024

I tested Livy 0.8.0 with Spark 3.2.4, using JDK 11, saw another failure, and have no idea ...

Caused by: java.lang.ClassCastException: class Bank cannot be cast to class Bank (Bank is in unnamed module of loader org.apache.spark.repl.ExecutorClassLoader @35007c02; Bank is in unnamed module of loader scala.tools.nsc.interpreter.IMain$TranslatingClassLoader @4dc4e5ff)

@Reamer
Copy link
Contributor Author

Reamer commented Oct 22, 2024

@pan3793 Thanks for #4880
I've adjusted this pull request.

@Reamer Reamer requested a review from pan3793 October 22, 2024 07:18
- python: 3.7
java: 8
- python: 3.8
java: 8
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this silently drops test for Python 3.7 and 3.8, if this is intended, please at least mention that in PR title/desc

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The description has been adjusted.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... also drops python 3.8 & 3.9

typo? should be 3.7 & 3.8

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely, thank you for your close look.
Adjusted.

Copy link
Member

@pan3793 pan3793 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, except the python version typo in desc

@Reamer Reamer merged commit b9860b6 into apache:master Oct 22, 2024
16 of 17 checks passed
asfgit pushed a commit that referenced this pull request Oct 22, 2024
### What is this PR for?
This pull request removes the test with JDK 8. I have kept the JDK matrix to make it easier to switch to the next JDK later.
This pull request also drops python 3.7 & 3.8 tests for the `python`,`rlang` and `zeppelin-jupyter` interpreter. Please note that current python 3.7 and 3.8 are end of life. https://devguide.python.org/versions/

### What type of PR is it?
Improvement

### What is the Jira issue?
 - https://issues.apache.org/jira/browse/ZEPPELIN-6090

### How should this be tested?
* CI

### Questions:
* Does the license files need to update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No

Closes #4875 from Reamer/jdk11_github.

Signed-off-by: Philipp Dallig <philipp.dallig@gmail.com>
(cherry picked from commit b9860b6)
Signed-off-by: Philipp Dallig <philipp.dallig@gmail.com>
@Reamer Reamer deleted the jdk11_github branch October 23, 2024 06:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants