Skip to content

Conversation

@markgrover
Copy link
Member

What changes were proposed in this pull request?

Allows to override locations for downloading Apache and Typesafe artifacts in build/mvn script.

How was this patch tested?

By running script like

# Remove all previously downloaded artifacts
rm -rf build/apache-maven*
rm -rf build/zinc-*
rm -rf build/scala-*

# Make sure path is clean and doesn't contain mvn, for example.
...

# Run a command without setting anything and make sure it succeeds
build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0 package

# Run a command setting the default location as mirror and make sure it succeeds
APACHE_MIRROR=http://mirror.infra.cloudera.com/apache/ build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0 package

# Do the same without the trailing slash this time and make sure it succeeds
APACHE_MIRROR=http://mirror.infra.cloudera.com/apache build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0 package

# Do it with a bad URL and make sure it fails
APACHE_MIRROR=xyz build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0 package

# Install maven under the build/ folder
install_mvn() {
local MVN_VERSION="3.3.9"
local APACHE_MIRROR=${APACHE_MIRROR:-https://archive.apache.org/dist}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't use a trailing slash in the definition of the default value here since I'd rather have 2 slashes at the end (if the user did define it) instead of no slash.

@markgrover markgrover changed the title [SPARK-SPARK-14477] [COMPONENT] Title: Allow custom mirrors for downloading artifacts in build/mvn [SPARK-14477] [COMPONENT] Title: Allow custom mirrors for downloading artifacts in build/mvn Apr 8, 2016
@markgrover markgrover changed the title [SPARK-14477] [COMPONENT] Title: Allow custom mirrors for downloading artifacts in build/mvn [SPARK-14477][BUILD] Allow custom mirrors for downloading artifacts in build/mvn Apr 8, 2016
install_app \
"https://downloads.typesafe.com/scala/${scala_version}" \
"${TYPESAFE_MIRROR}/scala/${scala_version}" \
"scala-${scala_version}.tgz" \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is unrelated to your change, but I wonder if Spark really needs to download the Scala distribution like this, or if the maven/ivy dependency resolution is enough.

@vanzin
Copy link
Contributor

vanzin commented Apr 8, 2016

LGTM.

@SparkQA
Copy link

SparkQA commented Apr 8, 2016

Test build #55284 has finished for PR 12250 at commit fcbcf26.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@vanzin
Copy link
Contributor

vanzin commented Apr 8, 2016

Unrelated test failure. Merging to master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants