HBase Shell Interpreter #278

vrajat · 2015-09-03T18:15:39Z

Support for Hbase Shell. All the commands documented here https://wiki.apache.org/hadoop/Hbase/Shell is supported.

Requirements:

Hbase Shell should be installed on the same machine. To be more specific, the following dir. should be available: https://github.com/apache/hbase/tree/master/hbase-shell/src/main/ruby
Hbase Shell should be able to connect to the Hbase cluster from terminal. This makes sure that the client is configured properly.
The interpreter takes 3 config parameters:

hbase.home: Root dir. where hbase is installed. Default is /usr/lib/hbase/
hbase.ruby.sources: Dir where shell ruby code is installed. Path is relative to hbase.home. Default: lib/ruby
hbase.irb.load: (Testing only) Default is true. Whether to load irb in the interpreter.

vrajat · 2015-09-03T23:30:44Z

Unit tests passed.

felixcheung · 2015-09-06T16:49:21Z

hbase/src/main/java/org/apache/zeppelin/hbase/HbaseInterpreter.java

if hbase.home hbase.ruby.sources are required for this interpreter would it make sense to have explicit checks for the hbase shell script file here?

That would make the error (in the case it is not presence) more explicit and more clear.

I already have a check. Refer to line 80.

To clarify, I'd suggest adding a check in code, not just logging the value - generally that is done in interpret() method instead of as one could return the error response to display to the user there.

Sorry. Completely missing the point here. I check if there is no hbase directory and I am throwing an InterpreterException. I am doing more than just logging the value. Can you point to examples ? Isnt it better to throw the exception in open method if the parameter is wrong ?

ok got it, sorry I think I was looking at line 84.
Could you use something like Paths or FilenameUtils instead of joining strings and requiring one has the path separator?

Done

On Mon, Jan 11, 2016 at 6:45 AM Felix Cheung notifications@github.com
wrote:

In hbase/src/main/java/org/apache/zeppelin/hbase/HbaseInterpreter.java
#278 (comment)
:

}

public HbaseInterpreter(Properties property) {

super(property);

}

@OverRide

public void open() {

String hbase_home = getProperty("hbase.home");

String ruby_src = getProperty("hbase.ruby.sources");

String abs_ruby_src = hbase_home + ruby_src;

logger.info("Home:" + hbase_home);

logger.info("Ruby Src:" + ruby_src);

Properties props = System.getProperties();

ok got it, sorry I think I was looking at line 84.
Could you use something like Paths
http://docs.oracle.com/javase/7/docs/api/java/nio/file/Paths.html or
FilenameUtils
https://commons.apache.org/proper/commons-io/javadocs/api-1.4/org/apache/commons/io/FilenameUtils.html
instead of joining strings and requiring one has the path separator?

—
Reply to this email directly or view it on GitHub
https://github.com/apache/incubator-zeppelin/pull/278/files#r49285311.

felixcheung · 2015-09-06T16:50:43Z

Could you elaborate on what this does?

hbase.irb.load: (Testing only) Default is true. Whether to load irb in the interpreter.

vrajat · 2015-09-07T20:08:23Z

@felixcheung I have addressed your comments. Please take a look

vrajat · 2015-09-10T20:22:23Z

The failure is a false positive. I made only cosmetic changes. Looks like it failed in downloading npm stuff.

felixcheung · 2015-09-12T00:42:11Z

hbase/src/main/java/org/apache/zeppelin/hbase/HbaseInterpreter.java

this should rethrow the exception too? I assume the interpreter will not work if this fails

This block is used for testing only. I'll add a comment.

bzz · 2016-01-05T10:04:35Z

@vrajat do you plan to update this PR\add some docs for new interpreter?
Or shall we consider closing it as it diverged from master and CI fails, so no way to merge it in the current state.

vrajat · 2016-01-05T11:55:40Z

I'll put some effort into clean up the PR.
On Tue, 5 Jan 2016 at 15:34, Alexander notifications@github.com wrote:

@vrajat https://github.com/vrajat do you plan to update this PR\add
some docs for new interpreter?
Or shall we consider closing it as it diverged from master and CI fails,
so no way to merge it in the current state.

—
Reply to this email directly or view it on GitHub
#278 (comment)
.

vrajat · 2016-01-21T04:22:13Z

@felixcheung Done

felixcheung · 2016-01-21T05:00:38Z

thanks, trying to test this now.
btw, this doc style update has just been committed - it would be great if you could check and conform to that, if not we could come back to that shortly.

felixcheung · 2016-01-21T05:11:08Z

also, I realize when testing this, you should add to the list in ZeppelinConfiguration.java as documented in here.
Sorry I didn't catch that earlier.

felixcheung · 2016-01-21T06:02:33Z

@vrajat I am having problem getting this to work. It's likely this is my setup but I wonder if you have any idea to fix this

HBase bin release from Apache Mirror hbase/hbase-1.0.2/
HBase running, hbase shell working
Built distribution for Zeppelin
set JAVA_HOME, start Zeppelin, enable hbase, set hbase.home in property

When I ran this

%hbase
help

I got this

(NameError) undefined local variable or method `help' for main:Object

and in the log

 INFO [2016-01-21 05:40:07,314] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:129) - Job remoteInterpretJob_1453354807312 started by scheduler org.apache.zeppelin.hbase.HbaseInterpreter599771689
 INFO [2016-01-21 05:40:07,494] ({pool-2-thread-2} HbaseInterpreter.java[open]:86) - Home:/opt/hbase-1.0.2
 INFO [2016-01-21 05:40:07,494] ({pool-2-thread-2} HbaseInterpreter.java[open]:87) - Ruby Src:lib/ruby
 INFO [2016-01-21 05:40:07,495] ({pool-2-thread-2} HbaseInterpreter.java[interpret]:110) - help
ERROR [2016-01-21 05:40:11,026] ({pool-2-thread-2} HbaseInterpreter.java[interpret]:117) - Can not run 'help'
org.jruby.embed.EvalFailedException: (NameError) undefined local variable or method `help' for main:Object
    at org.jruby.embed.internal.EmbedEvalUnitImpl.run(EmbedEvalUnitImpl.java:136)
    at org.jruby.embed.ScriptingContainer.runUnit(ScriptingContainer.java:1263)
    at org.jruby.embed.ScriptingContainer.runScriptlet(ScriptingContainer.java:1256)
    at org.apache.zeppelin.hbase.HbaseInterpreter.interpret(HbaseInterpreter.java:112)
    at org.apache.zeppelin.interpreter.ClassloaderInterpreter.interpret(ClassloaderInterpreter.java:57)
    at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:93)
    at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:305)
    at org.apache.zeppelin.scheduler.Job.run(Job.java:172)
    at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.jruby.exceptions.RaiseException: (NameError) undefined local variable or method `help' for main:Object

vrajat · 2016-01-21T06:04:49Z

Surprising. It seems like the hbase ruby scripts arent loaded. I used the
interpreter two days back. I'll try to reproduce the problem.

On Thu, Jan 21, 2016 at 11:32 AM Felix Cheung notifications@github.com
wrote:

@vrajat https://github.com/vrajat I am having problem getting this to
work. It's like this is my setup but I wonder if you have any idea to fix
this

HBase bin release from Apache Mirror hbase/hbase-1.0.2/

HBase running, hbase shell working

Built distribution for Zeppelin

set JAVA_HOME, start Zeppelin, enable hbase, set hbase.home in
property

When I ran this

%hbase
help

I got this

(NameError) undefined local variable or method `help' for main:Object

and in the log

INFO [2016-01-21 05:40:07,314]({pool-2-thread-2} SchedulerFactory.java[jobStarted]:129) - Job remoteInterpretJob_1453354807312 started by scheduler org.apache.zeppelin.hbase.HbaseInterpreter599771689
INFO [2016-01-21 05:40:07,494]({pool-2-thread-2} HbaseInterpreter.java[open]:86) - Home:/opt/hbase-1.0.2
INFO [2016-01-21 05:40:07,494]({pool-2-thread-2} HbaseInterpreter.java[open]:87) - Ruby Src:lib/ruby
INFO [2016-01-21 05:40:07,495]({pool-2-thread-2} HbaseInterpreter.java[interpret]:110) - help
ERROR [2016-01-21 05:40:11,026]({pool-2-thread-2} HbaseInterpreter.java[interpret]:117) - Can not run 'help'
org.jruby.embed.EvalFailedException: (NameError) undefined local variable or method help' for main:Object at org.jruby.embed.internal.EmbedEvalUnitImpl.run(EmbedEvalUnitImpl.java:136) at org.jruby.embed.ScriptingContainer.runUnit(ScriptingContainer.java:1263) at org.jruby.embed.ScriptingContainer.runScriptlet(ScriptingContainer.java:1256) at org.apache.zeppelin.hbase.HbaseInterpreter.interpret(HbaseInterpreter.java:112) at org.apache.zeppelin.interpreter.ClassloaderInterpreter.interpret(ClassloaderInterpreter.java:57) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:93) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:305) at org.apache.zeppelin.scheduler.Job.run(Job.java:172) at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.jruby.exceptions.RaiseException: (NameError) undefined local variable or methodhelp' for main:Object

—
Reply to this email directly or view it on GitHub
#278 (comment)
.

felixcheung · 2016-01-22T00:26:09Z

hbase shell is calling ${HBASE_HOME}/bin/hirb.rb - maybe this needs to do the same?

AhyoungRyu · 2016-01-22T03:15:36Z

@vrajat
Hi! It's really good to add HBase interpreter!!

I saw a documentation hbase.mdyou added. But currently, there is no link for this page. So you have to add a link to below this line. Then you can see your documentation page in the navigation tab : )

Plus, a few days ago, there is a PR #648 related to fixing documentation styles. This PR was working on deleting a numbering in the each sections. So why don't you remove a numbering?

Thank You ! : )

felixcheung · 2016-01-29T19:41:05Z

@vrajat ?

vrajat · 2016-01-30T01:59:38Z

Sorry. I've been busy with work. I'll try to get it next week. I'll look at
the bug. It must have been caused by one of the clean up commits.
With respect to docs, I will appreciate it if it's taken up later. This PR
has been going on for a year. I really want to get this done and move on.
On Sat, 30 Jan 2016 at 01:11, Felix Cheung notifications@github.com wrote:

@vrajat https://github.com/vrajat ?

—
Reply to this email directly or view it on GitHub
#278 (comment)
.

felixcheung · 2016-01-30T03:35:39Z

Sure - let's get it working, and here. I can help with updating doc if necessary.

vrajat · 2016-01-31T12:22:07Z

Ah ha! Now I remember why I had my own copy of hirb.rb. I couldnt get this piece of code in the original hirb.rb to work. So I had deleted it and it had no bad consequences.

35a36,41
> require 'pathname'
>
> # Add the directory names in hbase.jruby.sources commandline option
> # to the ruby load path so I can load up my HBase ruby modules
> sources = java.lang.System.getProperty('hbase.ruby.sources')
> $LOAD_PATH.unshift Pathname.new(sources)

felixcheung · 2016-02-01T07:00:09Z

@vrajat I see this error when I run it

 INFO [2016-02-01 06:55:28,887] ({pool-1-thread-3} HbaseInterpreter.java[open]:88) - Home:/opt/hbase-1.0.2/
 INFO [2016-02-01 06:55:28,887] ({pool-1-thread-3} HbaseInterpreter.java[open]:89) - Ruby Src:lib/ruby
 INFO [2016-02-01 06:55:28,887] ({pool-1-thread-3} HbaseInterpreter.java[open]:100) - Absolute Ruby Source:/opt/hbase-1.0.2/lib/ruby
ERROR [2016-02-01 06:55:28,904] ({pool-2-thread-2} Job.java[run]:184) - Job failed
org.apache.zeppelin.interpreter.InterpreterException: org.jruby.embed.EvalFailedException: (TypeError) can't convert Pathname into String
    at org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:76)
    at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:68)
    at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:92)
    at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:299)

But the path seems correct:

 ls /opt/hbase-1.0.2/lib/ruby
hbase  hbase.rb  irb  shell  shell.rb

Also there is a typo in zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java

felixcheung · 2016-02-01T07:00:53Z

zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java

this should be org.apache.zeppelin.hbase.HbaseInterpreter

vrajat · 2016-02-01T07:06:28Z

@felixcheung I've got past that error. I am hitting

java.lang.VerifyError: class org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$ColumnFamilySchema overrides final method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;
    at java.lang.ClassLoader.defineClass1(Native Method)

This usually implies a mismatch between the jars used in the hbase client and the hbase server. I am looking into it. The same code & dependencies works in the Qubole fork. So I am eliminating what could be different about this branch.

felixcheung · 2016-02-01T07:12:28Z

cool, let me know when I should try again.

vrajat · 2016-02-01T13:23:28Z

@felixcheung Please take a look now. I have fixed all bugs and I can use it on my HBase setup (HBase 1.0.0 and Hadoop 2.6.0)

felixcheung · 2016-02-02T05:55:43Z

Great! It works (tested with hbase-1.0.2).
Merging if no more discussion (I will help fix the doc format).

vrajat · 2016-02-02T05:56:27Z

w00t!

On Tue, Feb 2, 2016 at 11:25 AM Felix Cheung notifications@github.com
wrote:

Great! It works (tested with hbase-1.0.2).
Merging if no more discussion (I will help fix the doc format).

—
Reply to this email directly or view it on GitHub
#278 (comment)
.

Leemoonsoo · 2016-02-02T08:07:02Z

Cool!

fhygh · 2022-05-24T06:22:49Z

@felixcheung hello, i saw you using hbase-1.0.2 works fine, but i using hbase 2.4.9 got the same error:

INFO [2022-05-24 14:20:49,513] ({FIFOScheduler-org.apache.zeppelin.hbase.HbaseInterpreter679627819-Worker-1} HbaseInterpreter.java[open]:80) - Home:/data/ygh/hbase-2.4.9
INFO [2022-05-24 14:20:49,514] ({FIFOScheduler-org.apache.zeppelin.hbase.HbaseInterpreter679627819-Worker-1} HbaseInterpreter.java[open]:81) - Ruby Src:lib/ruby
INFO [2022-05-24 14:20:49,514] ({FIFOScheduler-org.apache.zeppelin.hbase.HbaseInterpreter679627819-Worker-1} HbaseInterpreter.java[open]:89) - Absolute Ruby Source:/data/ygh/hbase-2.4.9/lib/ruby
INFO [2022-05-24 14:20:50,252] ({FIFOScheduler-org.apache.zeppelin.hbase.HbaseInterpreter679627819-Worker-1} HbaseInterpreter.java[interpret]:115) -
help
ERROR [2022-05-24 14:20:50,262] ({FIFOScheduler-org.apache.zeppelin.hbase.HbaseInterpreter679627819-Worker-1} HbaseInterpreter.java[interpret]:122) - Can not run '
help'
org.jruby.embed.EvalFailedException: (NameError) undefined local variable or method help' for main:Object at org.jruby.embed.internal.EmbedEvalUnitImpl.run(EmbedEvalUnitImpl.java:136) at org.jruby.embed.ScriptingContainer.runUnit(ScriptingContainer.java:1263) at org.jruby.embed.ScriptingContainer.runScriptlet(ScriptingContainer.java:1256) at org.apache.zeppelin.hbase.HbaseInterpreter.interpret(HbaseInterpreter.java:117) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:110) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:860) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:752) at org.apache.zeppelin.scheduler.Job.run(Job.java:172) at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:132) at org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.jruby.exceptions.RaiseException: (NameError) undefined local variable or method help' for main:Object
INFO [2022-05-24 14:20:50,268] ({FIFOScheduler-org.apache.zeppelin.hbase.HbaseInterpreter679627819-Worker-1} AbstractScheduler.java[runJob]:154) - Job paragraph_1653298534725_1664890236 finished by scheduler org.apache.zeppelin.hbase.HbaseInterpreter679627819 with status ERROR

Below are the verisons :

Apache Zeppelin - 0.10.1
Hbase-2.4.9

do you have any idea to fix this?

Rajat Venkatesh added 17 commits April 29, 2015 10:10

Hbase Interpreter

bdba5b2

Fix test setup

03e1b7a

Add a profile for Hbase-1.0

cd41ec2

Add a new config for hbase version for interpreter

63a2d89

More fixes for version config vars

032a2a4

More fixes - sigh

b2ff198

Specify relative path

b9a407c

Merge remote branch 'upstream/master' into incubator_hbase

bc03f79

Fix version to add incubating string

b7b06cd

Add an explicit dependency for mapreduce-client-core

9a23a21

Merge master

afd50bc

Missed a couple of resolved conflicts

b8b52ba

Fix tab in pom.xml

feacd89

Explicitly add private keyword. Improve javadoc

c160787

Terminate jruby

19f137e

Fix javadoc in unit tests

b598b70

Add more info in javadoc

255d0d9

vrajat mentioned this pull request Sep 3, 2015

Hbase Interpreter #55

Closed

felixcheung reviewed Sep 6, 2015
View reviewed changes

Review comments

ca64931

felixcheung reviewed Sep 12, 2015
View reviewed changes

Merge from master

72306b7

Rajat Venkatesh added 6 commits January 31, 2016 12:38

Merge from master

49b5b27

Load hirb

b4ad18d

Address minor review comments

39dddb8

Fix hirb path

57ebad8

Set hbase.ruby.sources in system properties

69755ad

Log absolute path of ruby source

9924784

Do not use ruby 1.9 and do not bother setting paths

3efc892

felixcheung reviewed Feb 1, 2016
View reviewed changes

Fix typo in ZeppelinConfiguration

c17db3b

Fix Protobuf version number

caf87f3

asfgit closed this in 1940388 Feb 3, 2016

HBase Shell Interpreter #278

HBase Shell Interpreter #278

Uh oh!

Conversation

vrajat commented Sep 3, 2015

Uh oh!

vrajat commented Sep 3, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

felixcheung commented Sep 6, 2015

Uh oh!

vrajat commented Sep 7, 2015

Uh oh!

vrajat commented Sep 10, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bzz commented Jan 5, 2016

Uh oh!

vrajat commented Jan 5, 2016

Uh oh!

vrajat commented Jan 21, 2016

Uh oh!

felixcheung commented Jan 21, 2016

Uh oh!

felixcheung commented Jan 21, 2016

Uh oh!

felixcheung commented Jan 21, 2016

Uh oh!

vrajat commented Jan 21, 2016

Uh oh!

felixcheung commented Jan 22, 2016

Uh oh!

AhyoungRyu commented Jan 22, 2016

Uh oh!

felixcheung commented Jan 29, 2016

Uh oh!

vrajat commented Jan 30, 2016

Uh oh!

felixcheung commented Jan 30, 2016

Uh oh!

vrajat commented Jan 31, 2016

Uh oh!

felixcheung commented Feb 1, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vrajat commented Feb 1, 2016

Uh oh!

felixcheung commented Feb 1, 2016

Uh oh!

vrajat commented Feb 1, 2016

Uh oh!

felixcheung commented Feb 2, 2016

Uh oh!

vrajat commented Feb 2, 2016

Uh oh!

Leemoonsoo commented Feb 2, 2016

Uh oh!

fhygh commented May 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

fhygh commented May 24, 2022 •

edited

Loading