From 1f1d55172bbe4dc3fc3bef310ff86a679e321970 Mon Sep 17 00:00:00 2001 From: lizhengbing Date: Sat, 23 Aug 2014 11:04:41 +0800 Subject: [PATCH 1/5] Update building-with-maven.md Support building spark under http proxy environment --- docs/building-with-maven.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/building-with-maven.md b/docs/building-with-maven.md index a7d7bd3ccb1f2..0eda5f3246cc4 100644 --- a/docs/building-with-maven.md +++ b/docs/building-with-maven.md @@ -156,4 +156,12 @@ then ship it over to the cluster. We are investigating the exact cause for this. The assembly jar produced by `mvn package` will, by default, include all of Spark's dependencies, including Hadoop and some of its ecosystem projects. On YARN deployments, this causes multiple versions of these to appear on executor classpaths: the version packaged in the Spark assembly and the version on each node, included with yarn.application.classpath. The `hadoop-provided` profile builds the assembly without including Hadoop-ecosystem projects, like ZooKeeper and Hadoop itself. +# Building under http proxy environment +Sometimes,spark need be built in http proxy environment, We recommend the following settings: + + mvn -Pyarn -Phadoop-2.2 -Dhadoop.version=2.2.0 -Dmaven.wagon.http.ssl.insecure=true -Dmaven.wagon.http.ssl.allowall=true -DskipTests clean package + +If you don't run this, you may see errors like the following: + +[ERROR] Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:2.2:shade (default) on project spark-assembly_2.10: Execution default of goal From c41618510e8a6a42265b7591b890530cd0913a56 Mon Sep 17 00:00:00 2001 From: lizhengbing Date: Thu, 28 Aug 2014 10:48:44 +0800 Subject: [PATCH 2/5] Update building-with-maven.md --- docs/building-with-maven.md | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/docs/building-with-maven.md b/docs/building-with-maven.md index 0eda5f3246cc4..4e32fab7dc71b 100644 --- a/docs/building-with-maven.md +++ b/docs/building-with-maven.md @@ -158,10 +158,19 @@ The assembly jar produced by `mvn package` will, by default, include all of Spar # Building under http proxy environment -Sometimes,spark need be built in http proxy environment, We recommend the following settings: +Sometimes,spark is built in http-proxy environment. We recommend the following settings: + +
export MAVEN_OPTS="-Dmaven.wagon.http.ssl.insecure=true -Dmaven.wagon.http.ssl.allowall=true"
- mvn -Pyarn -Phadoop-2.2 -Dhadoop.version=2.2.0 -Dmaven.wagon.http.ssl.insecure=true -Dmaven.wagon.http.ssl.allowall=true -DskipTests clean package -If you don't run this, you may see errors like the following: +

If you don’t run this, you may see errors like the following:

+ +
Downloading: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-client/1.0.4/hadoop-client-1.0.4.pom
+Aug 27, 2014 5:08:21 PM org.apache.maven.wagon.providers.http.httpclient.client.protocol.RequestAuthenticationBase process
+WARNING: NTLM authentication error: Credentials cannot be used for NTLM authentication: org.apache.maven.wagon.providers.http.httpclient.auth.UsernamePasswordCredentials
+Downloading: https://repository.apache.org/content/repositories/releases/org/apache/hadoop/hadoop-client/1.0.4/hadoop-client-1.0.4.pom
+
+ +

You can fix this by setting the MAVEN_OPTS variable as discussed before.

-[ERROR] Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:2.2:shade (default) on project spark-assembly_2.10: Execution default of goal +

Note: the setting could lead to security risks of your Maven server.

From 51fd9ef869fab36772423a690079a23242f24327 Mon Sep 17 00:00:00 2001 From: lizhengbing Date: Thu, 28 Aug 2014 10:50:54 +0800 Subject: [PATCH 3/5] Update building-with-maven.md --- docs/building-with-maven.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/building-with-maven.md b/docs/building-with-maven.md index 4e32fab7dc71b..7235aca942dc6 100644 --- a/docs/building-with-maven.md +++ b/docs/building-with-maven.md @@ -156,7 +156,7 @@ then ship it over to the cluster. We are investigating the exact cause for this. The assembly jar produced by `mvn package` will, by default, include all of Spark's dependencies, including Hadoop and some of its ecosystem projects. On YARN deployments, this causes multiple versions of these to appear on executor classpaths: the version packaged in the Spark assembly and the version on each node, included with yarn.application.classpath. The `hadoop-provided` profile builds the assembly without including Hadoop-ecosystem projects, like ZooKeeper and Hadoop itself. -# Building under http proxy environment +# Building under http-proxy environment Sometimes,spark is built in http-proxy environment. We recommend the following settings: From 53a8caf18d7b578827407f96dbe45f8f699fe0d7 Mon Sep 17 00:00:00 2001 From: lizhengbing Date: Thu, 28 Aug 2014 10:53:23 +0800 Subject: [PATCH 4/5] Update building-with-maven.md --- docs/building-with-maven.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/building-with-maven.md b/docs/building-with-maven.md index 7235aca942dc6..79ef2ffc2be5a 100644 --- a/docs/building-with-maven.md +++ b/docs/building-with-maven.md @@ -169,6 +169,8 @@ Sometimes,spark is built in http-proxy environment. We recommend the following s Aug 27, 2014 5:08:21 PM org.apache.maven.wagon.providers.http.httpclient.client.protocol.RequestAuthenticationBase process WARNING: NTLM authentication error: Credentials cannot be used for NTLM authentication: org.apache.maven.wagon.providers.http.httpclient.auth.UsernamePasswordCredentials Downloading: https://repository.apache.org/content/repositories/releases/org/apache/hadoop/hadoop-client/1.0.4/hadoop-client-1.0.4.pom +Aug 27, 2014 5:08:21 PM org.apache.maven.wagon.providers.http.httpclient.client.protocol.RequestAuthenticationBase process +WARNING: NTLM authentication error: Credentials cannot be used for NTLM authentication: org.apache.maven.wagon.providers.http.httpclient.auth.UsernamePasswordCredentials

You can fix this by setting the MAVEN_OPTS variable as discussed before.

From 1aa816ba119bbf56bd6c6b21f9ba1ad2db5d8545 Mon Sep 17 00:00:00 2001 From: lizhengbing Date: Thu, 28 Aug 2014 10:55:25 +0800 Subject: [PATCH 5/5] Update building-with-maven.md --- docs/building-with-maven.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/building-with-maven.md b/docs/building-with-maven.md index 79ef2ffc2be5a..ec3831896f0d6 100644 --- a/docs/building-with-maven.md +++ b/docs/building-with-maven.md @@ -163,7 +163,7 @@ Sometimes,spark is built in http-proxy environment. We recommend the following s 
export MAVEN_OPTS="-Dmaven.wagon.http.ssl.insecure=true -Dmaven.wagon.http.ssl.allowall=true"
-

If you don’t run this, you may see errors like the following:

+

If you don’t run this, you may see warnings like the following:

Downloading: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-client/1.0.4/hadoop-client-1.0.4.pom
 Aug 27, 2014 5:08:21 PM org.apache.maven.wagon.providers.http.httpclient.client.protocol.RequestAuthenticationBase process