-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-36835][BUILD] Enable createDependencyReducedPom for Maven shaded plugin #34085
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Local build looks good, and let's see how Spark CI reacts. cc @JoshRosen |
|
Kubernetes integration test starting |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! I did a local mvn install and confirmed that the generated POMs correctly exclude the shaded depdendencies.
HyukjinKwon
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice!
|
Kubernetes integration test status failure |
|
Thanks, merging to master and branch-3.2 |
…ed plugin ### What changes were proposed in this pull request? Enable `createDependencyReducedPom` for Spark's Maven shaded plugin so that the effective pom won't contain those shaded artifacts such as `org.eclipse.jetty` ### Why are the changes needed? At the moment, the effective pom leaks transitive dependencies to downstream apps for those shaded artifacts, which potentially will cause issues. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? I manually tested and the `core/dependency-reduced-pom.xml` no longer contains dependencies such as `jetty-XX`. Closes #34085 from sunchao/SPARK-36835. Authored-by: Chao Sun <sunchao@apple.com> Signed-off-by: Gengliang Wang <gengliang@apache.org> (cherry picked from commit ed88e61) Signed-off-by: Gengliang Wang <gengliang@apache.org>
|
Test build #143574 has finished for PR 34085 at commit
|
|
+1, LGTM. |
|
@sunchao @JoshRosen I find that after merging this PR, Spark can't build with Hadoop2.7. The build hangs for over 1 hour Could you help fix it? |
|
@gengliangwang Oops, I'm taking a look on this. |
|
Yes this is the reason that last time I disabled the config. I think we are running into some Maven bug that is similar to this one. To fix it, I think we can put the newly introduced in |
|
I opened #34100 for this. |
What changes were proposed in this pull request?
Enable
createDependencyReducedPomfor Spark's Maven shaded plugin so that the effective pom won't contain those shaded artifacts such asorg.eclipse.jettyWhy are the changes needed?
At the moment, the effective pom leaks transitive dependencies to downstream apps for those shaded artifacts, which potentially will cause issues.
Does this PR introduce any user-facing change?
No
How was this patch tested?
I manually tested and the
core/dependency-reduced-pom.xmlno longer contains dependencies such asjetty-XX.