Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgrade spark client dependencies to be compatible with M1 #3420

Merged
merged 2 commits into from
May 24, 2022
Merged

Upgrade spark client dependencies to be compatible with M1 #3420

merged 2 commits into from
May 24, 2022

Conversation

lynnro314
Copy link
Contributor

@lynnro314 lynnro314 commented May 24, 2022 •
edited
Loading

bumped up testcontainers-scala, snappy-java and protoc version to be compatible with M1.

I used v.0.39.11 in testcontainers-scala and not the latest one since v.0.40.0 contains a breaking change.
With v.0.40.0 I got this error:

[error] /Users/lynnrozen/work/lakeFS/clients/spark/core/src/test/scala/io/treeverse/jpebble/BlockParserSpec.scala:308:7: type mismatch;
[error]  found   : (String, String, org.testcontainers.containers.BindMode)
[error]  required: com.dimafeng.testcontainers.GenericContainer.FileSystemBind
[error]       ("parser-test/sst_files_generator.go", "/local/sst_files_generator.go", BindMode.READ_WRITE),

@lynnro314 lynnro314 added the pr/merge-if-approved Reviewer: please feel free to merge if no major comments label May 24, 2022
@lynnro314 lynnro314 self-assigned this May 24, 2022
arielshaqed
arielshaqed approved these changes May 24, 2022
View reviewed changes
Copy link
Contributor

@arielshaqed arielshaqed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, thanks!

If you still have them handy, could you paste the errors from trying to upgrade to 0.40 (or whatever is latest)?

@lynnro314 lynnro314 added the exclude-changelog PR description should not be included in next release changelog label May 24, 2022
Jonathan-Rosenberg
Jonathan-Rosenberg reviewed May 24, 2022
View reviewed changes
clients/spark/build.sbt Outdated
@@ -64,11 +64,11 @@ def generateCoreProject(buildType: BuildType) =
// version will probably continue to work because the C language API
// is quite stable. Take the version documented in DataBricks
// Runtime 7.6, and note that it changes in 8.3 :-(
"org.xerial.snappy" % "snappy-java" % "1.1.7.5",
"org.xerial.snappy" % "snappy-java" % "1.1.8.2",
Copy link
Contributor

@Jonathan-Rosenberg Jonathan-Rosenberg May 24, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you try using 1.1.8.4?
It's not that I'm a big expert in snappy-java, it's just that the IDE suggested I change to 1.1.8.4 and it seemed to work...

@lynnro314 lynnro314 mentioned this pull request May 24, 2022
Closed
@lynnro314 lynnro314 merged commit 4fe7ae8 into master May 24, 2022
@lynnro314 lynnro314 deleted the update-spark-client-dependencies branch May 24, 2022 13:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Jonathan-Rosenberg Jonathan-Rosenberg Jonathan-Rosenberg left review comments

@arielshaqed arielshaqed arielshaqed approved these changes

Assignees

@lynnro314 lynnro314

Labels
exclude-changelog PR description should not be included in next release changelog pr/merge-if-approved Reviewer: please feel free to merge if no major comments
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@lynnro314 @arielshaqed @Jonathan-Rosenberg