[SPARK-54026][BUILD] Use BOM for AWS Java SDK V2 dependency management #52728
Conversation
| <dependency> | ||
| <groupId>software.amazon.awssdk</groupId> | ||
| <artifactId>bom</artifactId> | ||
| <version>${aws.java.sdk.v2.version}</version> |
There was a problem hiding this comment.
Since you have used BOM to manage versions, the explicit version definition at the sub-module's pom.xml is not required, right?
hadoop-cloud/pom.xml
resource-managers/kubernetes/integration-tests/pom.xml
There was a problem hiding this comment.
@pan3793 Both hadoop-cloud/pom.xml and resource-managers/kubernetes/integration-tests/pom.xml use bundle instead of individual jar dependencies while kinesis-asl relies on transitive dependencies of the producer library that use sts, auth and etc. BOM only defines individual dependencies and bundle still requires version to be specified in the dependency. Otherwise maven build fails:
./build/mvn -Pkubernetes-integration-tests -pl resource-managers/kubernetes/integration-tests
Using `mvn` from path: /local/home/vrozov/ws/spark-sbt/build/apache-maven-3.9.11/bin/mvn
[INFO] Maveniverse Nisse 0.4.6 loaded
[INFO] Nisse injecting 39 properties into User Properties
[INFO] Scanning for projects...
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[ERROR] 'dependencies.dependency.version' for software.amazon.awssdk:bundle:jar is missing. @ line 82, column 17
@
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR] The project org.apache.spark:spark-kubernetes-integration-tests_2.13:4.1.0-SNAPSHOT (/local/home/vrozov/ws/spark-sbt/resource-managers/kubernetes/integration-tests/pom.xml) has 1 error
[ERROR] 'dependencies.dependency.version' for software.amazon.awssdk:bundle:jar is missing. @ line 82, column 17
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
It may be necessary to see why both explicitly define dependency on bundle instead of relying on transitive dependencies. Though, I think, it is outside of the scope for this PR, WDYT?
| lazy val jacksonVersion = sys.props.get("fasterxml.jackson.version").getOrElse("2.20.0") | ||
| lazy val jacksonDeps = Bom.dependencies("com.fasterxml.jackson" % "jackson-bom" % jacksonVersion) | ||
| lazy val settings = jacksonDeps ++ Seq( | ||
| lazy val awsJavaSdkV2Version = sys.props.get("aws.java.sdk.v2.version").getOrElse("2.29.52") |
There was a problem hiding this comment.
the drawback is, we need to duplicate the version definition here again, would be better to investigate if we can extract the version from pom.xml directly
There was a problem hiding this comment.
Yes, I agree, it would be better to see if the property can be read from the pom.xml. As it needs to be done not only for AWS Java SDK v2, I'd suggest to do this on a separate PR.
|
@vrozov given the above two issues, I think this introduces complexity (manage one more dep, duplicate version definition in sbt config), but we can't benefit from that. I think we can re-evaluate this PR after addressing those two issues. |
|
@pan3793 The benefit is version consistency. Please see #52654 (comment).
This is an existing issue that applies to protobuf and jackson version. |
this is a new message to me, but it seems the issue only exists on branch-4.0? I'm afraid this patch won't launch branch-4.0
it's true, I had to add Jackson BOM and introduce the version duplication to unblock the Avro 1.12.1 upgrading, see #52664 (comment) @vrozov, to be clear, I don't oppose this change, but it does not look urgent, because I think it blocks nothing on the master branch. |
|
@pan3793 does the master branch accept only blocking changes?
I do not propose to backport this PR into branch-4.0 as it is an improvement and not a bug fix. I pointed to the 4.0 PR as an example where inconsistent AWS Java SDK version v2 leads to incorrect behavior. The goal of the change is to avoid inconsistency while managing AWS Java SDK v2 as explained in https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/setup-project-maven.html. |
2 participants
What changes were proposed in this pull request?
Use AWS Java SDK V2 BOM for consistent dependency management.
Why are the changes needed?
AWS Java SDK V2 expects consistent dependencies (https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/setup-project-maven.html)
Does this PR introduce any user-facing change?
No
How was this patch tested?
Checked Kinesis ASL works both with maven and SBT
Was this patch authored or co-authored using generative AI tooling?
No