ZEPPELIN-440 HiveInterpreter with multiple configuration #455

jongyoul · 2015-11-20T17:02:48Z

This PR enable HiveInterpreter to configure several connection information. For using specific configurations, you should use some tricks like %hive(configuration_prefix). without it, you can use default settings starting with default.

Leemoonsoo · 2015-11-21T10:50:14Z

Like the idea about '('.

jongyoul · 2015-11-22T06:04:23Z

@Leemoonsoo Thanks. If you have any new or more Zeppelinous idea for specifying properties, feel free to talk. I'll make it pass to CI and add docs for this function.

felixcheung · 2015-11-22T06:53:04Z

It's nice. Would it be possible to make the interpreter group/name fully customizable?
For a generic JDBC interpreter, it would be great if one could set to %hive and another to %presto, or %mysql

Leemoonsoo · 2015-11-22T09:43:58Z

User can multiple settings (profiles) for an interpreter, in Interpreter menu. For example,

spark-dev
spark-testing

settings can be made for the same %spark interpreter.

Current restriction is, only one interpreter setting can be used in the single notebook, in the same time.
That's because of selecting interpreter in paragraph is %[interpreter name], not %[setting name].

If we use %[setting name] for selecting interpreter,

Pros:

Allow multiple settings derived from single interpreter in a notebook.
User can set their favorite name for %[setting name] and use it in the paragraph

Cons:

When notebook file is shared to other people, it's hard to run while %[setting name] is customized and all Zeppelin instance will have all different setting name.

I think, in the future, we need to invent some way to allow all 'settings' being used in the same notebook at the same time, without having Cons that i mentioned above.

Until that time, I think this approach could be useful.

felixcheung · 2015-11-23T04:28:30Z

Good point - maybe we should save settings with notebook to make it portable?

jongyoul · 2015-11-23T09:16:07Z

@felixcheung First time, I also try to run interpreter by %{setting name}, but ... If it's possible to make notebook portable with settings, %{setting name} would be better.

jongyoul · 2015-11-23T09:59:48Z

trigger the CI again

jongyoul · 2015-11-23T10:22:29Z

travis fails while downloading dependencies

jongyoul · 2015-11-23T10:23:43Z

jongyoul · 2015-11-23T14:27:05Z

@Leemoonsoo Could you please check the travis log? I think this is about ZeppelinSparkClusterTest. Do you have any idea?

jongyoul · 2015-11-24T08:13:28Z

this failure depends on #468

jongyoul · 2015-11-25T04:48:30Z

I revised HiveInterpreter.executeSql from PostgresSqlInterpreter.executeSql. I think that would be more general way to implement

jongyoul · 2015-11-26T04:18:13Z

@Leemoonsoo @felixcheung Please review this PR, again. I've almost done. Additionally, @Leemoonsoo, after review, can I try to merge this PR into apache master branch by myself? If it's possible, could you please tell me a procedure for it?

bzz · 2015-11-26T04:35:08Z

@jongyoul merge a PR is very simple, you just:

setup git remote, named apache pointing to original apache repo https://git-wip-us.apache.org/repos/asf/incubator-zeppelin.git
run ./dev/merge_zeppelin_pr.py
(bonus): if you have envs JIRA_USERNAME and JIRA_PASSWORD setup - this script will take care of resolving the JIRA issue automatically

Leemoonsoo · 2015-11-26T04:37:00Z

About merging, you can use dev/merge_zeppelin_pr.py script.

I suggest clone zeppelin git repo into separate dir (separate from development) and then you'll need to add remote repo 'apache' and 'apache-github' to use this script.

apache  https://git-wip-us.apache.org/repos/asf/incubator-zeppelin (push)
apache  https://git-wip-us.apache.org/repos/asf/incubator-zeppelin (fetch)
apache-github   https://github.com/apache/incubator-zeppelin.git (fetch)
apache-github   https://github.com/apache/incubator-zeppelin.git (push)

Once it's done, just run dev/merge_zeppelin_pr.py will guide you merge the pull request.

bzz · 2015-11-26T04:40:51Z

hive/src/main/java/org/apache/zeppelin/hive/HiveInterpreter.java

What do you think, does horizontal alignment makes it more readable or just makes it longer?

According to the styleguide Zeppelin project uses - it is never required and has some cons https://google.github.io/styleguide/javaguide.html#s4.6.3-horizontal-alignment

Basically, I think it doesn't need except when that one line violates the maximum number of columns. I'll fix this, too.

bzz · 2015-11-26T04:46:19Z

👍 for test and docs!

jongyoul · 2015-11-27T06:40:21Z

@bzz Could you please verify this finally?

felixcheung · 2015-11-28T19:21:07Z

docs/interpreter/hive.md

typo: Zeppelin

parameterization

also use this relative path for the link (url has changed for multiple releases)
../manual/dynamicform.html

Or {{BASE_PATH}}/manual/dynamicform.html would work

felixcheung · 2015-11-28T19:22:34Z

docs/interpreter/hive.md

typo: wih -> with

Leemoonsoo · 2015-11-30T09:54:47Z

You maybe also want to change a link https://github.com/apache/incubator-zeppelin/blob/master/docs/_includes/themes/zeppelin/_navigation.html#L41

- Enable multiple connection properties

- Revert log4j.properties

- Remove commented codes - Fixed license part

- Fixed linefeed to variable

- Fixed style

- Fixed some tests

- Fixed getScriptBody return after trimming

- Restored some tests from the beginning

- Restored ParallelScheduler from FIFOScheduler

- Revised executeSql from Postgresql.executeSql

- Fixed test case to be more general

- Updated docs for hive

- Fixed Apaceh license header - Fixed some codestyle - Removed unused codes and classes

- Fixed typos on docs

- Rebased and fixed links of docs

jongyoul · 2015-11-30T10:16:30Z

@Leemoonsoo I've rebased and fixed it

jongyoul · 2015-12-01T06:39:15Z

Merging if there is no more discussion.

jongyoul changed the title ~~ZEPPELIN-440 HiveInterpreter with multiple configuration~~ [WIP]ZEPPELIN-440 HiveInterpreter with multiple configuration Nov 20, 2015

jongyoul force-pushed the ZEPPELIN-440 branch from e3a1f2a to 0fd5e7f Compare November 23, 2015 09:13

jongyoul closed this Nov 23, 2015

jongyoul reopened this Nov 23, 2015

jongyoul closed this Nov 23, 2015

jongyoul reopened this Nov 23, 2015

jongyoul closed this Nov 23, 2015

jongyoul reopened this Nov 23, 2015

jongyoul force-pushed the ZEPPELIN-440 branch from dc2803a to 6659c35 Compare November 24, 2015 06:22

jongyoul changed the title ~~[WIP]ZEPPELIN-440 HiveInterpreter with multiple configuration~~ ZEPPELIN-440 HiveInterpreter with multiple configuration Nov 26, 2015

bzz reviewed Nov 26, 2015
View reviewed changes

felixcheung reviewed Nov 28, 2015
View reviewed changes

docs/interpreter/hive.md Outdated

Copy link

Member

felixcheung Nov 28, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo: wih -> with

jongyoul added 16 commits November 30, 2015 18:59

ZEPPELIN-440 HiveInterpreter with multiple configuration

97bfa65

- Enable multiple connection properties

ZEPPELIN-440 HiveInterpreter with multiple configuration

87ee87a

- Revert log4j.properties

ZEPPELIN-440 HiveInterpreter with multiple configuration

df9f3cb

- Remove commented codes - Fixed license part

ZEPPELIN-440 HiveInterpreter with multiple configuration

9293d78

- Fixed linefeed to variable

ZEPPELIN-440 HiveInterpreter with multiple configuration

85e4914

- Fixed style

ZEPPELIN-440 HiveInterpreter with multiple configuration

8aa7601

- Fixed some tests

ZEPPELIN-440 HiveInterpreter with multiple configuration

5fa4fd7

- Fixed getScriptBody return after trimming

ZEPPELIN-440 HiveInterpreter with multiple configuration

2616050

- Restored some tests from the beginning

ZEPPELIN-440 HiveInterpreter with multiple configuration

f2b61c0

- Restored ParallelScheduler from FIFOScheduler

ZEPPELIN-440 HiveInterpreter with multiple configuration

dcb65ae

- Revised executeSql from Postgresql.executeSql

ZEPPELIN-440 HiveInterpreter with multiple configuration

4867c6f

- Fixed test case to be more general

ZEPPELIN-440 HiveInterpreter with multiple configuration

b4f1a1c

- Fixed test case to be more general

ZEPPELIN-440 HiveInterpreter with multiple configuration

4e01265

- Updated docs for hive

ZEPPELIN-440 HiveInterpreter with multiple configuration

fc33a30

- Fixed Apaceh license header - Fixed some codestyle - Removed unused codes and classes

ZEPPELIN-440 HiveInterpreter with multiple configuration

d68bcc5

- Fixed typos on docs

ZEPPELIN-440 HiveInterpreter with multiple configuration

af4c64b

- Rebased and fixed links of docs

jongyoul force-pushed the ZEPPELIN-440 branch from 6f55e3e to af4c64b Compare November 30, 2015 10:16

asfgit closed this in 7905939 Dec 4, 2015

jongyoul mentioned this pull request Dec 15, 2015

JDBC generic interpreter #361

Closed

ZEPPELIN-440 HiveInterpreter with multiple configuration #455

ZEPPELIN-440 HiveInterpreter with multiple configuration #455

Uh oh!

Conversation

jongyoul commented Nov 20, 2015

Uh oh!

Leemoonsoo commented Nov 21, 2015

Uh oh!

jongyoul commented Nov 22, 2015

Uh oh!

felixcheung commented Nov 22, 2015

Uh oh!

Leemoonsoo commented Nov 22, 2015

Uh oh!

felixcheung commented Nov 23, 2015

Uh oh!

jongyoul commented Nov 23, 2015

Uh oh!

jongyoul commented Nov 23, 2015

Uh oh!

jongyoul commented Nov 23, 2015

Uh oh!

jongyoul commented Nov 23, 2015

Uh oh!

jongyoul commented Nov 23, 2015

Uh oh!

jongyoul commented Nov 24, 2015

Uh oh!

jongyoul commented Nov 25, 2015

Uh oh!

jongyoul commented Nov 26, 2015

Uh oh!

bzz commented Nov 26, 2015

Uh oh!

Leemoonsoo commented Nov 26, 2015

Uh oh!

bzz Nov 26, 2015

Choose a reason for hiding this comment

Uh oh!

jongyoul Nov 26, 2015

Choose a reason for hiding this comment

Uh oh!

bzz commented Nov 26, 2015

Uh oh!

jongyoul commented Nov 27, 2015

Uh oh!

felixcheung Nov 28, 2015

Choose a reason for hiding this comment

Uh oh!

felixcheung Nov 28, 2015

Choose a reason for hiding this comment

Uh oh!

felixcheung Nov 28, 2015

Choose a reason for hiding this comment

Uh oh!

Leemoonsoo Nov 28, 2015

Choose a reason for hiding this comment

Uh oh!

felixcheung Nov 28, 2015

Choose a reason for hiding this comment

Uh oh!

Leemoonsoo commented Nov 30, 2015

Uh oh!

jongyoul commented Nov 30, 2015

Uh oh!

jongyoul commented Dec 1, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants