Skip to content

Presto Interpreter #68

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
danielhaviv wants to merge 2 commits into from
Closed

Presto Interpreter #68

danielhaviv wants to merge 2 commits into from

Conversation

@danielhaviv
Copy link

@danielhaviv danielhaviv commented May 11, 2015

Added an interpreter for Presto (http://prestodb.io)

@danielhaviv
Added a Presto interpreter
bbf10cd 
Added an interpreter for Presto (http://prestodb.io)
@Leemoonsoo
Copy link
Member

Leemoonsoo commented May 12, 2015

Thanks for the contribution! Great work!

One question. Is there any possibility to have a test using TestingPrestoServer.java? if it is not too difficult.

@danielhaviv
Copy link
Author

danielhaviv commented May 13, 2015

Yes
I will work on it

Daniel

On 13 במאי 2015, at 02:36, Lee moon soo notifications@github.com wrote:

Thanks for the contribution! Great work!

One question. Is there any possibility to have a test using TestingPrestoServer.java? if it is not too difficult.

—
Reply to this email directly or view it on GitHub.

Leemoonsoo
Leemoonsoo reviewed May 13, 2015
View reviewed changes
presto/pom.xml Outdated
Copy link
Member

@Leemoonsoo Leemoonsoo May 13, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

version should be 0.5.0-incubating-SNAPSHOT

@Leemoonsoo
Copy link
Member

Leemoonsoo commented May 31, 2015

Will there be some tests?

@Leemoonsoo
Copy link
Member

Leemoonsoo commented Jun 19, 2015

@danielhaviv Do you mind if i make PR to your branch that adds some tests?

@danielhaviv
Copy link
Author

danielhaviv commented Jun 19, 2015

Not at all.
My apologies for not having the time to take care of it.

Daniel

On 19 ביוני 2015, at 17:36, Lee moon soo notifications@github.com wrote:

@danielhaviv Do you mind if i make PR to your branch that adds some tests?

—
Reply to this email directly or view it on GitHub.

@danielhaviv
Copy link
Author

danielhaviv commented Jul 4, 2015

Hi,
I see the interpreter is currently not included but there's a Jira for it, any reason why ?

Thanks,
Daniel

@Leemoonsoo
Copy link
Member

Leemoonsoo commented Jul 9, 2015

Hi, @danielhaviv
Thanks for asking.

This interpreter code looks good but i think there should be some sort of test for it.
I recently i've been trying to implement some test for it, but i've stuck with JDK8 requirement.

It looks like version 0.87 and later requires JDK8 to use it's maven release artifact, while Zeppelin still stay with JDK7.

I'm trying to figure out the way, but any good idea about JDK8 requirement?

ashwinshankar77
ashwinshankar77 reviewed Aug 7, 2015
View reviewed changes
presto/src/main/java/org/apache/zeppelin/presto/PrestoInterpreter.java
Copy link

@ashwinshankar77 ashwinshankar77 Aug 7, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of hard-coding default, can we first try to get default through environment variables and system properties.
For eg. please see SparkInterpreter.getSystemDefault()
This would be useful because admins can set a defaults and users can just start using presto and not worry about updating properties on the interpreter page.

@ashwinshankar77
Copy link

ashwinshankar77 commented Aug 7, 2015

We need to add this interpreter in ZeppelinConfiguration.java under ZEPPELIN_INTERPRETERS,
otherwise zeppelin doesn't automatically show presto as one of the available interpreters on the UI.
Also add "presto" module to zeppelin pom.xml.

@martin-g
Copy link
Member

martin-g commented Aug 8, 2015

IMO the UI application should show only the discovered interpreters.
Each interpreter is in its own Maven module.
Zeppelin UI app should not load all interpreter modules with their transitive dependencies.
It should load only the ones me, as a developer, decided to put in the classpath.
Otherwise Zeppelin app soon will be ~1GB big and most users will use just the core and 1/2 interpreters, all the rest will be just extra unused stuff.

@jongyoul
Copy link
Member

jongyoul commented Nov 13, 2015

Linked to #361 because it has a same structure of a generic jdbc driver

@jongyoul
Copy link
Member

jongyoul commented Feb 25, 2016

@danielhaviv Do you mind closing this PR and updating doc for using Presto via JDBCInterpreter?

@danielhaviv
Copy link
Author

danielhaviv commented Feb 28, 2016

No longer relevant as there is now a JDBCInterpreter.

asfgit pushed a commit that referenced this pull request Dec 20, 2017
@kjmrknsn @Leemoonsoo
[ZEPPELIN-3077] Cron scheduler is easy to get stuck when one of the c…
888a05d 
…ron jobs takes long time or gets stuck

### What is this PR for?
The cron scheduler is easy to get stuck when one of the cron jobs takes long time or gets stuck.

I sometimes come across the issue that the cron scheduler stops working suddenly. According to the thread dump of ZeppelinServer, all of the DefaultQuartzScheduler_Worker threads were waiting for the job's completion and there was no thread to launch a new job.

Here is the contents of the thread dump:

```
"DefaultQuartzScheduler_Worker-10" #76 prio=5 os_prio=0 tid=0x00007fb41d3b4000 nid=0x1b521 sleeping[0x00007fb3daef1000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:889)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
        - locked <0x00000000c0a7dbf0> (a java.lang.Object)

   Locked ownable synchronizers:
        - None

"DefaultQuartzScheduler_Worker-9" #75 prio=5 os_prio=0 tid=0x00007fb41d3b2000 nid=0x1b520 waiting on condition [0x00007fb3daff2000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:889)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
        - locked <0x00000000c0a7a470> (a java.lang.Object)

   Locked ownable synchronizers:
        - None

...

"DefaultQuartzScheduler_Worker-2" #68 prio=5 os_prio=0 tid=0x00007fb41d3c8800 nid=0x1b519 waiting on condition [0x00007fb3da473000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:889)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
        - locked <0x00000000c0a7a7b0> (a java.lang.Object)

   Locked ownable synchronizers:
        - None

"DefaultQuartzScheduler_Worker-1" #67 prio=5 os_prio=0 tid=0x00007fb41d3cc800 nid=0x1b518 waiting on condition [0x00007fb3da372000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:889)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
        - locked <0x00000000c0a7dd90> (a java.lang.Object)

   Locked ownable synchronizers:
        - None
```

The above thread dump says that all of the worker threads get stuck at https://github.com/apache/zeppelin/blob/v0.7.3/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java#L889.

One way to reproduce this kind of issue is creating a paragraph whose status is "READY" and "disable run". That makes the paragraph status "READY" permanently and `note.isTerminated()` never turns to `true`.

To fix this issue, the following two improvements has been made at this PR:

1. Remove the unnecessary `while (!note.isTerminated()) { ... }` block because the execution of all of the paragraphs is finished after `note.runAll()`.
2. Skip the cron execution if there is a running or pending paragraph. That prevents the Zeppelin cron scheduler from getting stuck by the long running paragraph whose execution duration is greater than the cron execution cycle.

### What type of PR is it?
[Bug]

### Todos

### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-3077

### How should this be tested?
* Tested manually.
    1. The cron scheduler does not get stuck if there is a paragraph whose status is "READY" and "disable run".
    2. The following message is printed on the log file when the cron job is launched while the previous cron job still has been running.
        * `execution of the cron job is skipped because there is a running or pending paragraph (note id: XXXXXXXXX)`

### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? No.
* Is there breaking changes for older versions? No.
* Does this needs documentation? Yes. The behavior of the cron job was changed not to run if there is a running or pending paragraph by this PR. Thus, the documentation `docs/usage/other_features/cron_scheduler.md` was also added by this PR. Its layout is as follow:

<img width="711" alt="screen shot 2017-11-28 at 18 30 54" src="https://user-images.githubusercontent.com/31149688/33312407-20664e02-d46b-11e7-9715-9e2562d5e064.png">

Author: Keiji Yoshida <kjmrknsn@gmail.com>

Closes #2687 from kjmrknsn/ZEPPELIN-3077 and squashes the following commits:

81e7218 [Keiji Yoshida] [ZEPPELIN-3077] Cron scheduler is easy to get stuck when one of the cron jobs takes long time or gets stuck
jithinchandranj pushed a commit to jithinchandranj/zeppelin that referenced this pull request Dec 20, 2017
@kjmrknsn @jithinchandranj
[ZEPPELIN-3077] Cron scheduler is easy to get stuck when one of the c…
4adb2b9 
…ron jobs takes long time or gets stuck

### What is this PR for?
The cron scheduler is easy to get stuck when one of the cron jobs takes long time or gets stuck.

I sometimes come across the issue that the cron scheduler stops working suddenly. According to the thread dump of ZeppelinServer, all of the DefaultQuartzScheduler_Worker threads were waiting for the job's completion and there was no thread to launch a new job.

Here is the contents of the thread dump:

```
"DefaultQuartzScheduler_Worker-10" apache#76 prio=5 os_prio=0 tid=0x00007fb41d3b4000 nid=0x1b521 sleeping[0x00007fb3daef1000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:889)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
        - locked <0x00000000c0a7dbf0> (a java.lang.Object)

   Locked ownable synchronizers:
        - None

"DefaultQuartzScheduler_Worker-9" apache#75 prio=5 os_prio=0 tid=0x00007fb41d3b2000 nid=0x1b520 waiting on condition [0x00007fb3daff2000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:889)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
        - locked <0x00000000c0a7a470> (a java.lang.Object)

   Locked ownable synchronizers:
        - None

...

"DefaultQuartzScheduler_Worker-2" apache#68 prio=5 os_prio=0 tid=0x00007fb41d3c8800 nid=0x1b519 waiting on condition [0x00007fb3da473000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:889)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
        - locked <0x00000000c0a7a7b0> (a java.lang.Object)

   Locked ownable synchronizers:
        - None

"DefaultQuartzScheduler_Worker-1" apache#67 prio=5 os_prio=0 tid=0x00007fb41d3cc800 nid=0x1b518 waiting on condition [0x00007fb3da372000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:889)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
        - locked <0x00000000c0a7dd90> (a java.lang.Object)

   Locked ownable synchronizers:
        - None
```

The above thread dump says that all of the worker threads get stuck at https://github.com/apache/zeppelin/blob/v0.7.3/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java#L889.

One way to reproduce this kind of issue is creating a paragraph whose status is "READY" and "disable run". That makes the paragraph status "READY" permanently and `note.isTerminated()` never turns to `true`.

To fix this issue, the following two improvements has been made at this PR:

1. Remove the unnecessary `while (!note.isTerminated()) { ... }` block because the execution of all of the paragraphs is finished after `note.runAll()`.
2. Skip the cron execution if there is a running or pending paragraph. That prevents the Zeppelin cron scheduler from getting stuck by the long running paragraph whose execution duration is greater than the cron execution cycle.

### What type of PR is it?
[Bug]

### Todos

### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-3077

### How should this be tested?
* Tested manually.
    1. The cron scheduler does not get stuck if there is a paragraph whose status is "READY" and "disable run".
    2. The following message is printed on the log file when the cron job is launched while the previous cron job still has been running.
        * `execution of the cron job is skipped because there is a running or pending paragraph (note id: XXXXXXXXX)`

### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? No.
* Is there breaking changes for older versions? No.
* Does this needs documentation? Yes. The behavior of the cron job was changed not to run if there is a running or pending paragraph by this PR. Thus, the documentation `docs/usage/other_features/cron_scheduler.md` was also added by this PR. Its layout is as follow:

<img width="711" alt="screen shot 2017-11-28 at 18 30 54" src="https://user-images.githubusercontent.com/31149688/33312407-20664e02-d46b-11e7-9715-9e2562d5e064.png">

Author: Keiji Yoshida <kjmrknsn@gmail.com>

Closes apache#2687 from kjmrknsn/ZEPPELIN-3077 and squashes the following commits:

81e7218 [Keiji Yoshida] [ZEPPELIN-3077] Cron scheduler is easy to get stuck when one of the cron jobs takes long time or gets stuck
egorklimov pushed a commit to Tinkoff/zeppelin that referenced this pull request Sep 18, 2019
[ZP-000] Merge pull request apache#68 in DW/helicopter from ZP-000.co…
a0d857f 
…re.refactor.assetion_interface to V_1.0.0

* commit '63515cbfe88c785636fb545ae268e5d9a943f95e':
  [ZP-000] add assertion interface
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@danielhaviv @Leemoonsoo @ashwinshankar77 @martin-g @jongyoul