Failing to read META (hbase:meta)

[pfischermx]

Howdy,

I'm giving opentsdb a try. Our existing setup includes an experimentation hbase cluster (to play with) that runs hbase 0.98, we also use kerberos for authenticating with it.

I was able to create the tables and to configure tsdb to use the auth after I went with the latest RC (RC3), however I'm stuck in trying to start the tsdb, I keep getting the following:

2015-12-12 00:02:59,772 INFO [AsyncHBase Timer HBaseClient #1-SendThread()] ClientCnxn: Opening socket connection to server x.x.x.x:50512
2015-12-12 00:02:59,772 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-12 00:02:59,773 INFO [AsyncHBase Timer HBaseClient #1-SendThread(x.x.x.x:50512)] ClientCnxn: Socket connection established to x.x.x.x:50512, initiating session
2015-12-12 00:02:59,776 INFO [AsyncHBase Timer HBaseClient #1-SendThread(x.x.x.x:50512)] ClientCnxn: Session establishment complete on server x.x.x.x:50512, sessionid = 0x251705aced0233b, negotiated timeout = 5000
2015-12-12 00:02:59,778 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-12 00:02:59,779 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Probe Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=0, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key="")) failed
org.hbase.async.NonRecoverableException: Too many attempts: Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=11, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
at org.hbase.async.HBaseClient.tooManyAttempts(HBaseClient.java:2056) [asynchbase-1.7.1-20151004.015637-1.jar:na]
at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:1920) [asynchbase-1.7.1-20151004.015637-1.jar:na]
at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1944) [asynchbase-1.7.1-20151004.015637-1.jar:na]
at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1927) [asynchbase-1.7.1-20151004.015637-1.jar:na]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1005) [async-1.4.0.jar:na]
at org.hbase.async.HBaseClient$ZKClient$ZKCallback.processResult(HBaseClient.java:3632) [asynchbase-1.7.1-20151004.015637-1.jar:na]
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:565) [zookeeper-3.3.6.jar:3.3.6-1366786]
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) [zookeeper-3.3.6.jar:3.3.6-1366786]
2015-12-12 00:02:59,780 INFO [AsyncHBase I/O Worker #12] HBaseClient: Channel [id: 0x7c1125ef, /10.193.194.140:48266 :> /10.192.210.33:50511] is disconnecting: [id: 0x7c1125ef, /10.193.194.140:48266 :> /10.192.210.33:50511] DISCONNECTED

.. at the end it just dies with:

aused by: org.hbase.async.NonRecoverableException: Too many attempts: HBaseRpc(method=getClosestRowBefore, table="hbase:meta", key="perf_namespace:opentsdb,\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00:AsyncHBaseprobe<;<,:", region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""), attempt=11, timeout=0, hasTimedout=false)
at org.hbase.async.HBaseClient.tooManyAttempts(HBaseClient.java:2056)
at org.hbase.async.HBaseClient.handleNSRE(HBaseClient.java:2837)
... 29 more
Caused by: org.hbase.async.NotServingRegionException: Connection reset: [id: 0xaee6389e, /10.193.194.140:48389 :> /10.192.210.33:50511] got disconnected
Caused by RPC: HBaseRpc(method=getClosestRowBefore, table="hbase:meta", key="perf_namespace:opentsdb,\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00:AsyncHBaseprobe<;<,:", region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""), attempt=0, timeout=0, hasTimedout=false)

I can confirm (with hbase shell) that the table exists and can confirm that 10.129.210.33 is the region server holding hbase:root,,0.2132948379.

I'm using: asynchbase-1.7.1-20151004.015637-1.jar
hbase 0.98.7.14

Any ideas if I missed something or is this support for hbase?

thanks!

[manolama]

Hmm, a number of folks are seeing this. Are there any useful logs on the region server 10.129.210.33 at the time that you try to connect? And could you try setting the TSD in debug logging mode and pastebinning the sanitized output please?

[pfischermx]

Sorry for the delay, here is a full output of a tsd run.

https://gist.github.com/pfischermx/5c3c753db3bdb09efa76

(I needed to edit hostnames and a few other things).

Regarding checking the region server, I'll try but can't promise much here since that is managed by another team and gaining access to that data can take a while :/

Also, you mention other folks are seeing it, do they use kerberos?

[manolama]

Aha! Thanks for the Gist. Looks like you're at the Y! and using Grid's HBase. Ping me at clarsen and I can help you get up and running :) Our internal setup is a bit different than OS HBase.

[manolama]

Closing for now since it's an internal Yahoo issue :)

[wzktravel]

I met the same problem, how did you fix it?
asynchbase: 1.7.1
hbase: 1.0.0-cdh5.4.8

[karanalang]

Hello - i have the same issue .. pls let me know how to fix this.

thanks!

[waterazu]

Hi,

Check this [https://github.com//issues/491] in case you have a secured cluster.
In that case you can copy the jaas.conf file you use on you secured HBase, and make that file accessible to the JVM, in my case I added to the tsdb script the line:

-Djava.security.auth.login.config=/usr/share/opentsdb/bin/jaas.conf

so my last line of the script was now:

exec $JAVA $JVMARGS -Djava.security.auth.login.config=/usr/share/opentsdb/bin/jaas.conf -classpath "$CLASSPATH" net.opentsdb.tools.$MAINCLASS "$@"

Then I also had to use my Cloudera CDH 5.8.4 library of zookpeer.