Hadoop 2 woes

[nmaillard]

Hi everyone

I have installed cloudera wtith hadoop 2.04 so I compiled phoenix 1.2.1 with the hadoop 2 parameter. I don't know if anyone has tried this. I then tried a simple command line example and I keep getting the following error.
I guess this means somewhere there is still a hadoop 1.x running.

java.lang.NoSuchMethodError: org.apache.hadoop.net.NetUtils.getInputStream(Ljava/net/Socket;)Ljava/io/InputStream;
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:437)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1124)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:974)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy7.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HBaseAdmin.(HBaseAdmin.java:126)
at com.salesforce.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:517)

[ryang-sfdc]

I confirm a similar experience. Although the hadoop 2 profile in the phoenix pom pulls in hadoop 2 dependencies, it also pulls in a vanilla hbase 0.94.7 which itself was compiled against hadoop 1. :\

[jtaylor-sfdc]

Any ideas, @apurtell ?

[apurtell]

Until HBase puts up a Hadoop 2 artifact somewhere, you will have to build HBase against Hadoop 2 locally, so the artifacts you need are installed into the local Maven cache. Then you can use the Phoenix Hadoop 2 profile to build everything successfully.

That is HBASE-6929

[apurtell]

The Hadoop 2 profile in the Phoenix POM is to enable builds as part of constructing a Bigtop distribution. See BIGTOP-993. Everything works as expected in that environment because the target Hadoop version is specified throughout and packages are built in order of a topological sort of dependencies: First ZooKeeper, then Hadoop 2, then HBase (against Hadoop 2), then Phoenix, ...

[nmaillard]

Hi everyone

thanks for the help.
I kinda did like @apurtell said but instead of compiling against hadoop2 and adding to a convenient maven repo I went ahead and used the cloudera repo which has an hbase compiled againts hadoop2 namingly hbase0.94.6-cdh4.3. I added the repo and the dependency in the hadoop-2.

This makes for a little less work and effort.

Would it make sense for me to do a pull request for this change?
Would it make sense to go even further and actually have a cdh4.3 or cdhx profile for people using a given cloudera distribution. This is also valid for any other distribution.

[nmaillard]

I'll close this thread since a couple of solutions have been brought to the table