Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Spark 2.4 and Scala 2.12 #2044

Closed
heuermh opened this issue Sep 7, 2018 · 2 comments · Fixed by #2056
Closed

Support Spark 2.4 and Scala 2.12 #2044

heuermh opened this issue Sep 7, 2018 · 2 comments · Fixed by #2056
Milestone

Comments

@heuermh
Copy link
Member

heuermh commented Sep 7, 2018

The branch for Spark version 2.4 release has been cut, and it adds support for Scala 2.12

http://apache-spark-developers-list.1001551.n3.nabble.com/Branch-2-4-is-cut-td24928.html

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/job/spark-master-test-maven-hadoop-2.7-ubuntu-scala-2.12/229/

@antonkulaga
Copy link
Contributor

Spark 2.4 is officially releases already, looking forward to ADAMs Spark 2.4 support

@heuermh
Copy link
Member Author

heuermh commented Nov 12, 2018

Yep, it's going to take some effort. As you have seen, there is a blocker issue for us in Spark 2.4.0 upstream in Parquet 1.10.0 and how it interacts with Avro 1.8.2. The issue is not getting any attention, so we'll have to make some significant changes in ADAM, such as always reading via the Dataset path even if the caller is using the RDD path.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants