[SPARK-40331][DOCS] Recommend use Java 11/17 as the runtime environment of Spark #37799
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -40,6 +40,7 @@ source, visit [Building Spark](building-spark.html). | |
| Spark runs on both Windows and UNIX-like systems (e.g. Linux, Mac OS), and it should run on any platform that runs a supported version of Java. This should include JVMs on x86_64 and ARM64. It's easy to run locally on one machine --- all you need is to have `java` installed on your system `PATH`, or the `JAVA_HOME` environment variable pointing to a Java installation. | ||
|
|
||
| Spark runs on Java 8/11/17, Scala 2.12/2.13, Python 3.7+ and R 3.5+. | ||
| Java 11/17 is the recommended version to run Spark on. | ||
| Python 3.7 support is deprecated as of Spark 3.4.0. | ||
| Java 8 prior to version 8u201 support is deprecated as of Spark 3.2.0. | ||
|
Member
There was a problem hiding this comment. Hm, why do we remove this deprecation BTW?
Contributor
Author
There was a problem hiding this comment. I am ok to keep this line
Member
There was a problem hiding this comment. Yes, we should not remove this, @LuciferYang . Please revert this from this PR. |
||
| When using the Scala API, it is necessary for applications to use the same version of Scala that Spark was compiled for. | ||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder if we should explicitly mention this though. Some benchmark results weren't good actually IIRC.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I meant that the performance you mentioned looks limited to whole stage codegen and JIT complier only. I remember we saw some slower performance (e.g., at https://github.com/apache/spark/tree/master/core/benchmarks)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For compatibility, is the current code more inclined to the best practice of Java 8? Just as the current use of Scala 2.13 has not reached the best practice. I think it should be improved through some refactoring work, . @rednaxelafx , WDYT?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes we should move forward. But we should deprecate Java 8 first which I think would mean that encourage users to use JDK 11 and 17.
What I am unsure is that whether it's better to recommend something (JDK 11/17) slower when the old stuff (JDK 8) is not even deprecated. To end users, JDK 8 is still a faster option in general if I am not mistaken.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 for @HyukjinKwon 's comment. Yes, it does. Our benchmark shows JDK 8 is faster in general.