Skip to content

[SPARK-49988][BUILD] Remove unused Hadoop dependency management #48491

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 5 commits into from
Closed

[SPARK-49988][BUILD] Remove unused Hadoop dependency management #48491

wants to merge 5 commits into from

Conversation

pan3793
Copy link
Member

@pan3793 pan3793 commented Oct 16, 2024 •
edited
Loading

What changes were proposed in this pull request?

Remove unused vanilla hadoop dependency(and transitive deps) management, i.e. hadoop-client, xerces:xercesImpl, and inline deps defined in hadoop3 because it's the only supported hadoop profile.

Why are the changes needed?

Simplify pom.xml.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass CI and verified runtime jars are not affected by runningdev/test-dependencies.sh.

Was this patch authored or co-authored using generative AI tooling?

No.

@pan3793
Copy link
Member Author

pan3793 commented Oct 16, 2024

cc @dongjoon-hyun @LuciferYang

LuciferYang
LuciferYang reviewed Oct 16, 2024
View reviewed changes
pom.xml
-->
<dependency>
<groupId>xerces</groupId>
<artifactId>xercesImpl</artifactId>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it because Apache Spark doesn't require the xerces:xercesImpl dependency?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If that's the case, should we clean up the corresponding binary LICENSE of xerces:xercesImpl ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's a transitive dep of hadoop vanilla client, spark does not directly use it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right, updated LICENSE and NOTICE

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be best to declare this change in the pr description.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated

LuciferYang
LuciferYang approved these changes Oct 16, 2024
View reviewed changes
Copy link
Contributor

@LuciferYang LuciferYang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM(pending tests)

dongjoon-hyun
dongjoon-hyun reviewed Oct 16, 2024
View reviewed changes
Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you. Is this tested with Maven too, @pan3793 ?

@pan3793
Copy link
Member Author

pan3793 commented Oct 17, 2024

@dongjoon-hyun I tested the maven build, run dev/test-dependencies.sh to ensure that runtime deps are affected, and just run UT of a few modules(sts, yarn) using maven locally.

@LuciferYang
Copy link
Contributor

We can add the following code to build_and_test.yml to have GA execute a Maven test once.

maven-test:
    name: "Run Maven Test"
    permissions:
      packages: write
    uses: ./.github/workflows/maven_test.yml

@github-actions github-actions bot added the INFRA label Oct 17, 2024
@pan3793
Copy link
Member Author

pan3793 commented Oct 17, 2024

@LuciferYang thanks for the information, I trigger a Maven build and test on CI, https://github.com/pan3793/spark/actions/runs/11379852474, let's wait for the results.

@pan3793
Revert "run maven test"
85d78e9 
This reverts commit 41d365e.
@github-actions github-actions bot removed the INFRA label Oct 17, 2024
dongjoon-hyun
dongjoon-hyun approved these changes Oct 17, 2024
View reviewed changes
Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM. Thank you, @pan3793 and @LuciferYang .
Merged to master for Apache Spark 4.0.0 on February 2025.

himadripal pushed a commit to himadripal/spark that referenced this pull request Oct 19, 2024
@pan3793
[SPARK-49988][BUILD] Remove unused Hadoop dependency management
8ef53ac 
### What changes were proposed in this pull request?

Remove unused vanilla hadoop dependency(and transitive deps) management, i.e. `hadoop-client`, `xerces:xercesImpl`, and inline deps defined in `hadoop3` because it's the only supported hadoop profile.

### Why are the changes needed?

Simplify pom.xml.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass CI and verified runtime jars are not affected by running`dev/test-dependencies.sh`.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#48491 from pan3793/SPARK-49988.

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@LuciferYang LuciferYang LuciferYang approved these changes

@dongjoon-hyun dongjoon-hyun dongjoon-hyun approved these changes

Assignees
No one assigned
Labels
BUILD KUBERNETES
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@pan3793 @LuciferYang @dongjoon-hyun