Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update pyspark 2.4.4 to pyspark 3.0.0 #2678

Closed
schrockn opened this issue Jul 3, 2020 · 3 comments
Closed

Update pyspark 2.4.4 to pyspark 3.0.0 #2678

schrockn opened this issue Jul 3, 2020 · 3 comments

Comments

@schrockn
Copy link
Member

schrockn commented Jul 3, 2020

We should upgrade to the newest spark version in our integration image and tests
@natekupp
Copy link
Contributor

natekupp commented Jul 3, 2020

see also #1960

noting here for whoever looks into this, IIRC spark 2/3 have different Java and debian version requirements, so this may be a bit involved to fix.

@natekupp
Copy link
Contributor

natekupp commented Jul 3, 2020

see http://spark.apache.org/docs/latest/

Spark runs on Java 8/11, Scala 2.12, Python 2.7+/3.4+ and R 3.1+. Java 8 prior to version 8u92 support is deprecated as of Spark 3.0.0. Python 2 and Python 3 prior to version 3.6 support is deprecated as of Spark 3.0.0. R prior to version 3.4 support is deprecated as of Spark 3.0.0. For the Scala API, Spark 3.0.0 uses Scala 2.12. You will need to use a compatible Scala version (2.12.x).

@natekupp
Copy link
Contributor

natekupp commented Aug 4, 2020

Fixed by https://dagster.phacility.com/D4091

@natekupp natekupp closed this as completed Aug 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@natekupp @schrockn