Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BLOCKING][CI] Upgrade to Spark 2.4.3 #4414

Merged
merged 11 commits into from
May 10, 2019
Merged

[BLOCKING][CI] Upgrade to Spark 2.4.3 #4414

merged 11 commits into from
May 10, 2019

Conversation

hcho3
Copy link
Collaborator

@hcho3 hcho3 commented Apr 27, 2019 •
edited by CodingCat
Loading

See #4409. Spark 2.4.1 has a serious bug.

closes #4409.

@hcho3 hcho3 requested a review from CodingCat April 27, 2019 21:14
CodingCat
CodingCat approved these changes Apr 27, 2019
View reviewed changes
Copy link
Member

@CodingCat CodingCat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe you can update pom.xml as well?

@hcho3
Copy link
Collaborator Author

hcho3 commented Apr 27, 2019

@CodingCat The SparkTraining example doesn't seem to be compatible with Spark 2.4.2. Can you take a look?
https://xgboost-ci.net/blue/organizations/jenkins/xgboost/detail/PR-4414/8/pipeline/204#step-245-log-3381

@CodingCat
Copy link
Member

it's strange, do we cache some previously built pkg?

@CodingCat
Copy link
Member

ok, I checked 2.4.2 released tar ball

looks like we are affected due to the release issue in 2.4.2, as 2.4.2 mistakenly released with scala 2.12 in their tar ball....

let's wait for 2.4.3..........

@CodingCat CodingCat changed the title [CI] Upgrade to Spark 2.4.2 [BLOCKING][CI] Upgrade to Spark 2.4.3 Apr 28, 2019
@hcho3
Copy link
Collaborator Author

hcho3 commented Apr 28, 2019 •
edited
Loading

@CodingCat

as 2.4.2 mistakenly released with scala 2.12 in their tar ball....

Was that a mistake? The download page says that "Spark is pre-built with Scala 2.12 since version 2.4.2"

@CodingCat
Copy link
Member

Yeah, they are not supposed to change scala version in a maintenance release, leading to 2.4.1.tar.gz is for scala 2.11 but 2.4.2.tar.gz is for 2.12

@CodingCat
Copy link
Member

2.4.3 is ready, when ahead to modify this PR and see how it goes

@CodingCat CodingCat merged commit 6ff9941 into dmlc:master May 10, 2019
@hcho3 hcho3 deleted the upgrade_spark branch May 10, 2019 04:41
@lock lock bot locked as resolved and limited conversation to collaborators Aug 8, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@CodingCat CodingCat CodingCat approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[BLOCKING][jvm-packages] upgrade and test with spark 2.4.3
2 participants
@hcho3 @CodingCat