Error in loading storage handler.org.yong3.hive.mongo.MongoStorageHandler

[cooldude12]

Hi,
Thanks for the hive plug in. I ran into some issue while following readme to test the installation.
I am getting error in loading the storage handler. It may be a path or some environment variable issue.

I verified : <mongo_hive_home>/src/main/java/org/yong3/hive/mongo/MongoStorageHandler.java exists.

mongo_hive home dir: /home/ben/tools/mongo_hive

here is the full log

[DEV] ben@bi-all:~/tools$ $HIVE_HOME/bin/hive --auxpath /home//ben/tools/mongo_hive/mongo-java-driver-2.6.3.jar,/home/ben/tools/mongo_hive/guava-r06.jar, /home/ben/tools/mongo_hive/hive-mongo-0.0.1-SNAPSHOT.jar
Logging initialized using configuration in jar:file:/home/ben/tools/hive-0.8.1-bin/lib/hive-common-0.8.1.jar!/hive-log4j.properties
Hive history file=/tmp/ben/hive_job_log_ben_201207211824_1059679816.txt
hive> create external table mongo_events(id int, dt int, ip string, event string) stored by "org.yong3.hive.mongo.MongoStorageHandler" with serdeproperties ( "mongo.column.mapping" = "_id,dt,ip,event" ) tblproperties ( "mongo.host" = "bi-fb-app.dev", "mongo.port" = "27017", "mongo.db" = "game", "mongo.collection" = "event_20120703" );
Failed with exception org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.org.yong3.hive.mongo.MongoStorageHandler
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

[yc-huang]

could you help to post what's the error message in your hive log (which is located in /tmp//hive.log)?

thx,
YC

[cooldude12]

YC,
Thanks for the reply. I think I missed the step of building all the jar files. Since I don't have a java back ground here are my questions:

1. Do I need to build all thrre jar files? or can download it. I am running ubuntu
   mongo-java-driver-2.6.3.jar, guava-r06.jar, hive-mongo-0.0.1-SNAPSHOT.jar or I can download first two of them?
   Do you have a documentation on the all the build of each of this?

I see you have some answer for the snapshot: #3. Is the snapshot.jar is for files which you have created [i.e under mongo_hive/src/main/**...

for guava*jatr, I downloaded guava-r06-site.jar, will that work? or i need to do complete build after downloading all the source files

for mongo java driver, i downloaded the tar files. How do i make jar out of it.
mongodb-mongo-java-driver-674c521

1. After the building jar is completed, should i be able to do just use your readme doc? do in eed to set any other env variables?
2. What are the limitation of existing package. Does it support group by etc.
3. Any opinion on mongo-hadoop package. Do you know when 10gen is going to support hive. We are storing good amount of data in mongo, and all we care about is running hive. I suppose it will be faster than the regular hadoop+hive as mongo is faster on map-reduce than hadoop. Do you think so?

thanks a lot for your help

[cooldude12]

since you asked for hive log, here is it

012-07-23 00:24:33,293 WARN conf.HiveConf (HiveConf.java:(68)) - hive-site.xml not found on CLASSPATH
2012-07-23 00:24:38,765 ERROR exec.Task (SessionState.java:printError(380)) - Failed with exception org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.org.yong3.hive.mongo.MongoStorageHandler
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.org.yong3.hive.mongo.MongoStorageHandler
at org.apache.hadoop.hive.ql.metadata.Table.getStorageHandler(Table.java:278)
at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3360)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:225)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:133)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1332)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1123)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.org.yong3.hive.mongo.MongoStorageHandler
at org.apache.hadoop.hive.ql.metadata.HiveUtils.getStorageHandler(HiveUtils.java:151)
at org.apache.hadoop.hive.ql.metadata.Table.getStorageHandler(Table.java:273)
... 17 more
Caused by: java.lang.ClassNotFoundException: org.yong3.hive.mongo.MongoStorageHandler
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.hive.ql.metadata.HiveUtils.getStorageHandler(HiveUtils.java:144)
... 18 more

2012-07-23 00:24:38,765 ERROR ql.Driver (SessionState.java:printError(380)) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
2012-07-23 00:34:26,314 WARN conf.HiveConf (HiveConf.java:(68)) - hive-site.xml not found on CLASSPATH
2012-07-23 00:34:29,517 ERROR exec.Task (SessionState.java:printError(380)) - Failed with exception org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.org.yong3.hive.mongo.MongoStorageHandler
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.org.yong3.hive.mongo.MongoStorageHandler
at org.apache.hadoop.hive.ql.metadata.Table.getStorageHandler(Table.java:278)
at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3360)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:225)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:133)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1332)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1123)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
"hive.log" 111L, 8379C

[yc-huang]

Sorry for miss your email. Now I include a combo jar under 'target' directory you might try without needing guava/mongo-java-driver jar...