Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RFC: Removing Spark 1.x, Scala 2.10 support in 0.24.0 release #1597

Closed
fnothaft opened this issue Jul 10, 2017 · 4 comments
Closed

RFC: Removing Spark 1.x, Scala 2.10 support in 0.24.0 release #1597

fnothaft opened this issue Jul 10, 2017 · 4 comments
Labels
discussion
Milestone
0.24.0

Comments

@fnothaft
Copy link
Member

From the spark-dev list, it looks like Spark 2.3.0 will add support for Scala 2.12 and drop support for Scala 2.10. As such, I think that we should drop Spark 1.x and Scala 2.10 support in the first ADAM release after Spark 2.3.0, which for now, I am assuming to be ADAM 0.24.0. Thoughts?
@fnothaft fnothaft added this to the 0.24.0 milestone Jul 10, 2017
@heuermh
Copy link
Member

heuermh commented Jul 11, 2017

+1

For me, the stragglers are cgcloud and dockerized Spark for Toil, both of which are still our responsibility to some extent. I haven't heard from any external users still wanting Spark 1.x.

@antonkulaga
Copy link
Contributor

+1

@antonkulaga
Copy link
Contributor

Maybe for 0.23 release, you can at least make Spark 2 and Scala 2.11 the default one so people will not need to run 2.11/spark2 bash scripts each time the clone adam?

@heuermh
Copy link
Member

heuermh commented Dec 5, 2017

Maybe for 0.23 release, you can at least make Spark 2 and Scala 2.11 the default one...

Sorry, misread this originally. We don't want to change the release scripts before the 0.23.0 release, so Spark 1.x/Scala 2.10 will remain the default.

@heuermh heuermh mentioned this issue Dec 28, 2017
Merged
@heuermh heuermh mentioned this issue Jan 5, 2018
Merged
9 tasks
@fnothaft fnothaft mentioned this issue Jan 5, 2018
Merged
fnothaft pushed a commit that referenced this issue Jan 9, 2018
@heuermh @fnothaft
[ADAM-1597] Move to Scala 2.11 and Spark 2.x.
90572b5 
Resolves #1597:

* Remove Scala 2.10 and Spark 1.x from jenkins-test.
* Only build and release Scala 2.11 and Spark 2.x.
* Bump Scala dependency version to 2.11.12.
* Bump Spark dependency version to 2.2.1.
* Use Spark version 2.2.1 in jenkins-test.
* Try curl with Location redirect for Apache mirror script.
* Remove mention of Scala 2.10 and Spark 1.x from docs.
* Remove references to cgcloud.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
discussion
Projects
None yet
Milestone
0.24.0
Development

No branches or pull requests
3 participants
@heuermh @antonkulaga @fnothaft