use hadoop 2.8.0-palantir2 #107

sjrand · 2017-02-19T16:51:30Z

@robert3005 @pwoody @ash211 for real this time

List of differences between upstream branch-2.8.0 and 2.8.0-palantir1: https://github.com/palantir/hadoop/blob/branch-2.8.0/PALANTIR-CHANGELOG.md

sjrand · 2017-02-19T17:32:16Z

In addition to the tests that ran as part of the build (https://circleci.com/gh/palantir/hadoop/156), I also ran the s3a tests on my laptop (pointed to an actual s3 bucket).

The only failures were that you can overwrite a directory, which is expected based on having reverted HADOOP-13188. (The revert is hopefully temporary, but causes us to have the same behavior as we already have in 2.7.3, so not a regression.)

sjrand · 2017-02-20T00:31:33Z

Grr. Need to pull in hadoop-hdfs JAR to deal with

Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider not found
  at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2122)
  at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2214)

Actual fix is probably on the Hadoop side, not the Spark side -- filed https://issues.apache.org/jira/browse/HDFS-11431.

… the classes it relies on

sjrand · 2017-02-20T01:41:56Z

Tried briefly to fix HDFS-11431, but it's a bit icky -- the ConfiguredFailoverProxyProvider class calls a bunch of other code from hadoop-hdfs, so it's not trivial to bring it into hadoop-hdfs-client. For now just added a commit to bring in hadoop-hdfs, which is not a regression relative to 2.7.3

sjrand · 2017-02-20T14:18:50Z

Apparently I've now broken a bunch of hive tests. Can't tell whether this is the cause, but looks relevant:

WARN org.apache.spark.sql.hive.client.IsolatedClientLoader: Failed to resolve Hadoop artifacts for the version 2.8.0-palantir1. We will change the hadoop version from 2.8.0-palantir1 to 2.4.0 and try again. Hadoop classes will not be shared between Spark and Hive metastore client. It is recommended to set jars used by Hive metastore client through spark.sql.hive.metastore.jars in the production environment.

sjrand · 2017-02-25T18:05:00Z

Hive test failures are evidently caused by inability to resolve the Hadoop version.

2017-02-19 22:05:56.12 - stderr> 		[FAILED     ] org.apache.hadoop#hadoop-client;2.8.0-palantir1!hadoop-client.jar:  (0ms)
2017-02-19 22:05:56.12 - stderr> 
2017-02-19 22:05:56.12 - stderr> 	==== local-m2-cache: tried
2017-02-19 22:05:56.12 - stderr> 
2017-02-19 22:05:56.12 - stderr> 	  file:/home/ubuntu/spark/dummy/.m2/repository/org/apache/hadoop/hadoop-client/2.8.0-palantir1/hadoop-client-2.8.0-palantir1.jar
2017-02-19 22:05:56.12 - stderr> 
2017-02-19 22:05:56.12 - stderr> 	==== local-ivy-cache: tried
2017-02-19 22:05:56.12 - stderr> 
2017-02-19 22:05:56.12 - stderr> 	  /home/ubuntu/.ivy2/local/org.apache.hadoop/hadoop-client/2.8.0-palantir1/jars/hadoop-client.jar
2017-02-19 22:05:56.12 - stderr> 
2017-02-19 22:05:56.12 - stderr> 	==== central: tried
2017-02-19 22:05:56.12 - stderr> 
2017-02-19 22:05:56.121 - stderr> 	  https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client/2.8.0-palantir1/hadoop-client-2.8.0-palantir1.jar
2017-02-19 22:05:56.121 - stderr> 
2017-02-19 22:05:56.121 - stderr> 	==== spark-packages: tried
2017-02-19 22:05:56.121 - stderr> 
2017-02-19 22:05:56.121 - stderr> 	  http://dl.bintray.com/spark-packages/maven/org/apache/hadoop/hadoop-client/2.8.0-palantir1/hadoop-client-2.8.0-palantir1.jar
2017-02-19 22:05:56.121 - stderr> 
2017-02-19 22:05:56.121 - stderr> 	==== repo-1: tried
2017-02-19 22:05:56.121 - stderr> 
2017-02-19 22:05:56.121 - stderr> 	  http://www.datanucleus.org/downloads/maven2/org/apache/hadoop/hadoop-client/2.8.0-palantir1/hadoop-client-2.8.0-palantir1.jar

robert3005 · 2017-02-25T18:06:33Z

Why the penultimate isn't a hit?

…

On Sat, 25 Feb 2017 at 19:05, sjrand ***@***.***> wrote: Hive test failures are evidently caused by inability to resolve the Hadoop version. 2017-02-19 22:05:56.12 - stderr> [FAILED ] org.apache.hadoop#hadoop-client;2.8.0-palantir1!hadoop-client.jar: (0ms) 2017-02-19 22:05:56.12 - stderr> 2017-02-19 22:05:56.12 - stderr> ==== local-m2-cache: tried 2017-02-19 22:05:56.12 - stderr> 2017-02-19 22:05:56.12 - stderr> file:/home/ubuntu/spark/dummy/.m2/repository/org/apache/hadoop/hadoop-client/2.8.0-palantir1/hadoop-client-2.8.0-palantir1.jar 2017-02-19 22:05:56.12 - stderr> 2017-02-19 22:05:56.12 - stderr> ==== local-ivy-cache: tried 2017-02-19 22:05:56.12 - stderr> 2017-02-19 22:05:56.12 - stderr> /home/ubuntu/.ivy2/local/org.apache.hadoop/hadoop-client/2.8.0-palantir1/jars/hadoop-client.jar 2017-02-19 22:05:56.12 - stderr> 2017-02-19 22:05:56.12 - stderr> ==== central: tried 2017-02-19 22:05:56.12 - stderr> 2017-02-19 22:05:56.121 - stderr> https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client/2.8.0-palantir1/hadoop-client-2.8.0-palantir1.jar 2017-02-19 <https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client/2.8.0-palantir1/hadoop-client-2.8.0-palantir1.jar2017-02-19> 22:05:56.121 - stderr> 2017-02-19 22:05:56.121 - stderr> ==== spark-packages: tried 2017-02-19 22:05:56.121 - stderr> 2017-02-19 22:05:56.121 - stderr> http://dl.bintray.com/spark-packages/maven/org/apache/hadoop/hadoop-client/2.8.0-palantir1/hadoop-client-2.8.0-palantir1.jar 2017-02-19 <http://dl.bintray.com/spark-packages/maven/org/apache/hadoop/hadoop-client/2.8.0-palantir1/hadoop-client-2.8.0-palantir1.jar2017-02-19> 22:05:56.121 - stderr> 2017-02-19 22:05:56.121 - stderr> ==== repo-1: tried 2017-02-19 22:05:56.121 - stderr> 2017-02-19 22:05:56.121 - stderr> http://www.datanucleus.org/downloads/maven2/org/apache/hadoop/hadoop-client/2.8.0-palantir1/hadoop-client-2.8.0-palantir1.jar — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#107 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAfQVLFAxKudHlFeUMrVeis4fHidgvq1ks5rgG1NgaJpZM4MFg7G> .

robert3005 · 2017-02-25T18:07:28Z

You need to add our bintray. Won't accept with failing tests

…

On Sat, 25 Feb 2017 at 19:06, Robert Kruszewski ***@***.***> wrote: Why the penultimate isn't a hit? On Sat, 25 Feb 2017 at 19:05, sjrand ***@***.***> wrote: Hive test failures are evidently caused by inability to resolve the Hadoop version. 2017-02-19 22:05:56.12 - stderr> [FAILED ] org.apache.hadoop#hadoop-client;2.8.0-palantir1!hadoop-client.jar: (0ms) 2017-02-19 22:05:56.12 - stderr> 2017-02-19 22:05:56.12 - stderr> ==== local-m2-cache: tried 2017-02-19 22:05:56.12 - stderr> 2017-02-19 22:05:56.12 - stderr> file:/home/ubuntu/spark/dummy/.m2/repository/org/apache/hadoop/hadoop-client/2.8.0-palantir1/hadoop-client-2.8.0-palantir1.jar 2017-02-19 22:05:56.12 - stderr> 2017-02-19 22:05:56.12 - stderr> ==== local-ivy-cache: tried 2017-02-19 22:05:56.12 - stderr> 2017-02-19 22:05:56.12 - stderr> /home/ubuntu/.ivy2/local/org.apache.hadoop/hadoop-client/2.8.0-palantir1/jars/hadoop-client.jar 2017-02-19 22:05:56.12 - stderr> 2017-02-19 22:05:56.12 - stderr> ==== central: tried 2017-02-19 22:05:56.12 - stderr> 2017-02-19 22:05:56.121 - stderr> https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client/2.8.0-palantir1/hadoop-client-2.8.0-palantir1.jar 2017-02-19 <https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client/2.8.0-palantir1/hadoop-client-2.8.0-palantir1.jar2017-02-19> 22:05:56.121 - stderr> 2017-02-19 22:05:56.121 - stderr> ==== spark-packages: tried 2017-02-19 22:05:56.121 - stderr> 2017-02-19 22:05:56.121 - stderr> http://dl.bintray.com/spark-packages/maven/org/apache/hadoop/hadoop-client/2.8.0-palantir1/hadoop-client-2.8.0-palantir1.jar 2017-02-19 <http://dl.bintray.com/spark-packages/maven/org/apache/hadoop/hadoop-client/2.8.0-palantir1/hadoop-client-2.8.0-palantir1.jar2017-02-19> 22:05:56.121 - stderr> 2017-02-19 22:05:56.121 - stderr> ==== repo-1: tried 2017-02-19 22:05:56.121 - stderr> 2017-02-19 22:05:56.121 - stderr> http://www.datanucleus.org/downloads/maven2/org/apache/hadoop/hadoop-client/2.8.0-palantir1/hadoop-client-2.8.0-palantir1.jar — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#107 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAfQVLFAxKudHlFeUMrVeis4fHidgvq1ks5rgG1NgaJpZM4MFg7G> .

sjrand · 2017-02-25T18:11:19Z

Yep, fair enough. Trying to find where to add it -- I'm a little confused because we already include http://dl.bintray.com/palantir/releases in our pom

…cies.sh --replace-manifest

This reverts commit 358041a.

This reverts commit b3e99ad.

sjrand · 2017-02-26T03:04:59Z

Woo, finally got a green build. I'm running another build now with the hadoop version bumped from 2.8.0-palantir1 to 2.8.0-palantir2, which picks up some upstream s3a fixes. If that build also passes, are you guys down to merge this?

robert3005 · 2017-02-27T15:00:02Z

core/pom.xml

      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-client</artifactId>
    </dependency>
+    <dependency>


Why is this necessary now?

If I don't have this dependency then the resulting dist doesn't have hadoop-hdfs-2.8.0-palantir2.jar in it jars/ dir.

Ok. Something has changed in the packaging since with 2.7.3 it's there. Would be good to understand what happened upstream but not blocking

I think it's https://issues.apache.org/jira/browse/HDFS-6200. Previously hadoop-client would bring in hadoop-hdfs, but after that change, hadoop-client brings in hadoop-hdfs-client instead. But then because of https://issues.apache.org/jira/browse/HDFS-11431, I have to manually add hadoop-hdfs back in.

robert3005 · 2017-02-27T15:00:49Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala

        hiveArtifacts.mkString(","),
        SparkSubmitUtils.buildIvySettings(
-          Some("http://www.datanucleus.org/downloads/maven2"),
+          Some("http://dl.bintray.com/palantir/releases"),


Can this be a list? (not sure what's the exact signature). Fine if not

Apparently yes: @param remoteRepos Comma-delimited string of remote repositories other than maven central Will fix

robert3005 · 2017-02-27T15:01:05Z

pom.xml

          </exclusion>
        </exclusions>
      </dependency>
+      <!-- TODO (srand) Remove this when https://issues.apache.org/jira/browse/HDFS-11431 is fixed -->


Can you create an issue as well and reference it here

robert3005 · 2017-02-27T15:01:19Z

couple of pom nits, otherwise 👍

ash211 · 2017-02-28T03:34:58Z

@sjrand just to confirm, we expect a client running hadoop-2.8.0-palantir2 to work with a server of hadoop-2.7.3 ? And what about the reverse: a client of 2.7.3 against a hadoop-palantir server?

ash211 · 2017-02-28T04:10:32Z

Merging this PR caused this failure in the Circle test:

 4877 [info] ExternalSorterSuite:
 4878 [info] - empty data stream with kryo ser (79 milliseconds)
<snip>
 4888 [info] - cleanup of intermediate files in sorter (154 milliseconds)
 4889 [info] - cleanup of intermediate files in sorter with failures (141 milliseconds)
 4890 [info] - cleanup of intermediate files in shuffle (286 milliseconds)
 4891 [info] - cleanup of intermediate files in shuffle with failures *** FAILED *** (96 milliseconds)
 4892 [info]   java.lang.AssertionError: assertion failed: expected test shuffle cleanup to spill, but did not
 4893 [info]   at scala.Predef$.assert(Predef.scala:170)
 4894 [info]   at org.apache.spark.TestUtils$.assertSpilled(TestUtils.scala:178)
 4895 [info]   at org.apache.spark.util.collection.ExternalSorterSuite.org$apache$spark$util$collection$ExternalSorterSuite$$cleanupIntermediateFilesInShuffle(ExternalSorterSuite.scala:503)
 4896 [info]   at org.apache.spark.util.collection.ExternalSorterSuite$$anonfun$4.apply$mcV$sp(ExternalSorterSuite.scala:63)
 4897 [info]   at org.apache.spark.util.collection.ExternalSorterSuite$$anonfun$4.apply(ExternalSorterSuite.scala:63)
 4898 [info]   at org.apache.spark.util.collection.ExternalSorterSuite$$anonfun$4.apply(ExternalSorterSuite.scala:63)
 4899 [info]   at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
 4900 [info]   at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
 4901 [info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
 4902 [info]   at org.scalatest.Transformer.apply(Transformer.scala:22)

I think it's a flake -- https://circleci.com/gh/palantir/spark/424 -- so kicked off another build.

sjrand · 2017-02-28T14:07:01Z

Yes, a Spark client application running 2.8.0-palantir2 against a 2.7* (or vendored equivalent) cluster has worked fine in my experience. MapReduce clients have not fared so well (classpath stuff), but that's not relevant here.

There are no plans to run palantir-hadoop on the cluster -- no reason to try to play Hadoop vendor when several other companies already do a way better job of it than I could.

* Change the API contract for uploading local jars. This mirrors similarly to what YARN and Mesos expects. * Address comments * Fix test

sjrand added 2 commits February 19, 2017 11:45

use hadoop 2.8.0-palantir1

424230a

name resulting distribution more descriptively

3e68d1b

bring in hadoop-hdfs to get ConfiguredFailoverProxyProvider class and…

2422fa8

… the classes it relies on

sjrand added 2 commits February 19, 2017 21:41

exclude commons-daemon and update hadoop profile of tests

a12a3d8

more exclusions. damn you hadoop

5f02ec7

update hadoop profile in test-dependencies.sh

5d590df

sjrand force-pushed the sr/palantir-hadoop-2.8.0-palantir1 branch from e9e3871 to 5d590df Compare February 25, 2017 18:59

sjrand added 6 commits February 25, 2017 14:03

commit spark-deps-hadoop-palantir file from running dev/test-dependen…

6d66b97

…cies.sh --replace-manifest

note that we should drop the dep on hadoop-hdfs when HDFS-11431 is fixed

f04e624

datanucleus -> bintray

358041a

Revert "datanucleus -> bintray"

b3e99ad

This reverts commit 358041a.

bump to 2.8.0-palantir2

7ad085e

Revert "Revert "datanucleus -> bintray""

9bbc2be

This reverts commit b3e99ad.

sjrand added 2 commits February 25, 2017 22:09

banish spark-deps-hadoop-2.7 to /dev/null

1f58be9

bump timeout

74b53c5

sjrand changed the title ~~use hadoop 2.8.0-palantir1~~ use hadoop 2.8.0-palantir2 Feb 27, 2017

robert3005 reviewed Feb 27, 2017

View reviewed changes

robert3005 closed this Feb 27, 2017

robert3005 reopened this Feb 27, 2017

list of maven repos for hive client loader

443348e

robert3005 merged commit a22ccff into master Feb 27, 2017

robert3005 deleted the sr/palantir-hadoop-2.8.0-palantir1 branch February 27, 2017 16:07

robert3005 pushed a commit that referenced this pull request Mar 7, 2017

Change the API contract for uploading local files (#107)

22a2e5a

* Change the API contract for uploading local jars. This mirrors similarly to what YARN and Mesos expects. * Address comments * Fix test

mccheah added a commit that referenced this pull request Apr 27, 2017

Change the API contract for uploading local files (#107)

6a999ca

* Change the API contract for uploading local jars. This mirrors similarly to what YARN and Mesos expects. * Address comments * Fix test

use hadoop 2.8.0-palantir2 #107

use hadoop 2.8.0-palantir2 #107

Uh oh!

Conversation

sjrand commented Feb 19, 2017

Uh oh!

sjrand commented Feb 19, 2017

Uh oh!

sjrand commented Feb 20, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sjrand commented Feb 20, 2017

Uh oh!

sjrand commented Feb 20, 2017

Uh oh!

sjrand commented Feb 25, 2017

Uh oh!

robert3005 commented Feb 25, 2017 via email

Uh oh!

robert3005 commented Feb 25, 2017 via email

Uh oh!

sjrand commented Feb 25, 2017

Uh oh!

sjrand commented Feb 26, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

robert3005 commented Feb 27, 2017

Uh oh!

ash211 commented Feb 28, 2017

Uh oh!

ash211 commented Feb 28, 2017

Uh oh!

sjrand commented Feb 28, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sjrand commented Feb 20, 2017 •

edited

Loading