[SPARK-10359] Enumerate Spark's dependencies in a file and diff against it for new pull requests #8531
Conversation
|
Jenkins, test this please. |
|
Test build #41816 has finished for PR 8531 at commit
|
|
Jenkins, retest this please. |
|
Jenkins, test this please. |
|
Test build #41817 has finished for PR 8531 at commit
|
|
Test build #41818 has finished for PR 8531 at commit
|
|
Given that we just had a build break in only one of the Hadoop branches due to a dep. change, I'm thinking that this script should generate four dependencies files, one for each Hadoop profile that we test on Jenkins. |
Conflicts: dev/run-tests-jenkins dev/sparktestsupport/modules.py
|
Test build #44874 has started for PR 8531 at commit |
|
Test build #44876 has finished for PR 8531 at commit
|
|
Jenkins, test this please. |
pwendell
force-pushed
the
dependency-audits
branch
from
d5d98f4 to
2f4d3e5
Compare
|
Jenkins, test this please. |
|
Test build #44886 has finished for PR 8531 at commit
|
|
Jenkins, retest this please. |
|
Test build #44889 has finished for PR 8531 at commit
|
|
Test build #44895 has finished for PR 8531 at commit
|
pwendell
force-pushed
the
dependency-audits
branch
from
ca686a3 to
37230f0
Compare
There was a problem hiding this comment.
Yep, I think you're right. According to the maven-dependency-plugin docs (https://maven.apache.org/plugins/maven-dependency-plugin/list-mojo.html#includeScope):
Scope to include. An Empty string indicates all scopes (default). The scopes being interpreted are the scopes as Maven sees them, not as specified in the pom. In summary:
- runtime scope gives runtime and compile dependencies,
- compile scope gives compile, provided, and system dependencies,
- test (default) scope gives all dependencies,
- provided scope just gives provided dependencies,
- system scope just gives system dependencies.
Based on this language, it seems like we want to be using runtime here so that we include both compile and runtime scope dependencies but do not exclude provided ones.
|
Test build #44907 has finished for PR 8531 at commit
|
|
Jenkins, test this please. |
|
Test build #44950 has finished for PR 8531 at commit
|
|
Are you sure that we should sort the classpath? If two JARs provide the same classes then the ordering might matter and the relative classpath ordering might change as a consequence of adding a new dependency. Could also leave that up to followup, though, since this patch is massively useful as-is. |
There was a problem hiding this comment.
I'm thinking of taking over this PR and am considering dropping this logic since it adds complexity and might not be a huge performance issue in practice. Let me know if you disagree.
…r new pull requests This patch adds a new build check which enumerates Spark's resolved runtime classpath and saves it to a file, then diffs against that file to detect whether pull requests have introduced dependency changes. The aim of this check is to make it simpler to reason about whether pull request which modify the build have introduced new dependencies or changed transitive dependencies in a way that affects the final classpath. This supplants the checks added in SPARK-4123 / apache#5093, which are currently disabled due to bugs. This patch is based on pwendell's work in apache#8531. Closes apache#8531. Author: Josh Rosen <joshrosen@databricks.com> Author: Patrick Wendell <patrick@databricks.com> Closes apache#10461 from JoshRosen/SPARK-10359.
…r new pull requests This patch adds a new build check which enumerates Spark's resolved runtime classpath and saves it to a file, then diffs against that file to detect whether pull requests have introduced dependency changes. The aim of this check is to make it simpler to reason about whether pull request which modify the build have introduced new dependencies or changed transitive dependencies in a way that affects the final classpath. This supplants the checks added in SPARK-4123 / apache#5093, which are currently disabled due to bugs. This patch is based on pwendell's work in apache#8531. Closes apache#8531. Author: Josh Rosen <joshrosen@databricks.com> Author: Patrick Wendell <patrick@databricks.com> Closes apache#10461 from JoshRosen/SPARK-10359.
4 participants
DON'T MERGE ME - TESTING ON JENKINS