Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature Request]: Support more recent versions of Spark 3 #22156

Closed
mosche opened this issue Jul 5, 2022 · 0 comments
Closed

[Feature Request]: Support more recent versions of Spark 3 #22156

mosche opened this issue Jul 5, 2022 · 0 comments
Assignees
@mosche
Labels
done & done Issue has been reviewed after it was closed for verification, followups, etc. java P2 runners spark

Comments

@mosche
Copy link
Member

mosche commented Jul 5, 2022

What would you like to happen?

The Spark 3 runner is currently compiled using Spark 3.1.2 and fails using laters versions.
Compilation and tests should be fixed to work with Spark 3.2 and 3.3 as well.

As far as known this currently mostly affects the metrics integration (and the structured streaming runner).

Note: Spark 3.2/3.2 also bump their Avro version to 1.10/1.11. This may cause further issues if Avro is used. See
#19969 and potentially other issues

Issue Priority

Priority: 2

Issue Component

Component: runner-spark
@mosche mosche self-assigned this Jul 5, 2022
mosche pushed a commit to mosche/beam that referenced this issue Jul 5, 2022
Fixes apache#22156: Fix Spark3 runner to compile against Spark 3.2/3.…
72b1134 
…3 and add version tests to verify compatibility going forward.
@mosche mosche mentioned this issue Jul 6, 2022
Closed
@mosche mosche closed this as completed in 72127f9 Jul 22, 2022
@chamikaramj chamikaramj added the done & done Issue has been reviewed after it was closed for verification, followups, etc. label Oct 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mosche mosche

Labels
done & done Issue has been reviewed after it was closed for verification, followups, etc. java P2 runners spark
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mosche @chamikaramj