ZEPPELIN-963 ] Jobmanagement - (1) basic backend. #972

cloverhearts · 2016-06-07T09:02:41Z

What is this PR for?

Job management basic backend.
was divided into smaller " PR ". - #921
Receive the basic data of Job manager api and the backend have been implemented.

What type of PR is it?

Feature

Todos

- Basic backend for job manager.
- Basic backend api

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-963

How should this be tested?

step 1.

First, calling the rest api as follows receives the data of "job" of the whole.

curl -H "Content-Type: application/json" -X GET http://127.0.0.1:8080/api/notebook/jobmanager/

result

{
   "status":"OK", // result for api request
   "body":{
      "lastResponseUnixTime":1465289938763, // last get server unixtime stamp.
      "jobs":[  // job list
         {
            "notebookId":"2BMQZ9QP6",  // notebook id.
            "unixTimeLastRun":1465289017310, // notebook last running unixtime.
            "notebookType":"normal", // cron or normal
            "isRunningJob":false, // is Running?
            "notebookName":"Untitled Note 1226", // notebook name.
            "interpreter":"spark", // default interpreter group name or If you have not selected it does not exist.
            "paragraphs":[
               {
                  "name":"20160607-174331_232775609", // paragraph name 'undefined is notebook id'
                  "id":"20160607-174331_232775609", // paragraph id
                  "status":"FINISHED" // paragraph job status
               }
            ]
         }
      ]
   }
}

step 2.

For example, it showed the result of receiving the information as one of the notebook.
focus on "lastResponseUnixTime" value.
This value is inserted as an argument when you call the following restapi, us to be able to get the updated data.

Create a Notebook, or run the Paragraph.
And, call updated notebook api

"lastResponseUnixTime": 1465289938763 (step 1 get value)

curl -H "Content-Type: application/json" -X GET http://127.0.0.1:8080/api/notebook/jobmanager/1465289938763

step 3.

If you created a Notebook, a new Notebook information is displayed.
If there is a Notebook that if there is a Paragraph of the running, the information will also be displayed.

step 4.

When you restart the step1 of api, if you created a Notebook, the number of Notebook is +1 than when the first call.

Questions:

Does the licenses files need update? no
Is there breaking changes for older versions? no
Does this needs documentation? yes

cloverhearts · 2016-06-07T10:53:41Z

retry ci

jongyoul · 2016-06-07T13:36:34Z

Thanks. Do you have any test scenario? It would help to add test cases.

cloverhearts · 2016-06-07T14:22:52Z

@jongyoul Thank you for your feedback.
I'll write about test scenario.

jongyoul · 2016-06-07T14:48:04Z

How about changing this title to add ZEPPELIN-963?

cloverhearts · 2016-06-07T14:52:39Z

@jongyoul I've changed.

cloverhearts · 2016-06-08T13:20:52Z

Update text for 'How should this be tested' and add json data comment.

astroshim · 2016-06-08T14:46:24Z

It's working nicely.
but Is the correct behavior that the lastResponseUnixTime is increased whenever call the curl?
Then i can't understand what's the purpose of step 3. Could you tell me more detail?

cloverhearts · 2016-06-08T15:01:24Z

@astroshim

http://127.0.0.1:8080/api/notebook/jobmanager/(get unixtime at before step)

for example

http://127.0.0.1:8080/api/notebook/jobmanager/
(get unixtime is 1)

http://127.0.0.1:8080/api/notebook/jobmanager/1
(get unixtime is 2)
http://127.0.0.1:8080/api/notebook/jobmanager/2
....

cloverhearts · 2016-06-09T02:23:54Z

Updated text for test scenario.

@astroshim
Yes, lastResponseUnixTime value is update data area key.
to query the Future updates based on LastResponseUnixTime value by updating continue this value,
it is possible to get the data updated based on the lastResponseUnixTime.

astroshim · 2016-06-09T02:34:37Z

@cloverhearts Thank you for your explanation. You mean that the "LastResponseUnixTime" is for getting notebook information of lastest updated.

cloverhearts · 2016-06-09T05:10:35Z

Please review

jongyoul · 2016-06-09T13:38:06Z

zeppelin-server/src/main/java/org/apache/zeppelin/rest/NotebookRestApi.java

+    List<Map<String, Object>> notebookJobs = notebook.getJobListforNotebook(false, 0);
+    Map<String, Object> response = new HashMap<>();
+
+    response.put("lastResponseUnixTime", System.currentTimeMillis());


There's actual time gap between System.currentTimeMillis and the result of notebook.getJobListforNotebook. Is it OK? Or I think you can use some information of notebookJobs.

Of course, exists a gap. But the gap is very small.
In addition, there is a solution for it.
'UnixTimeLastRun' value jobs in the Notebook will display the last update time.
It sends a 'unixTimeLastRun' value to the update request.
You can receive data without a gap of time since the update of the Notebook.

@cloverhearts If this is intended behaviour and isn't influenced on the right behaviour, I'm OK.

cloverhearts · 2016-06-13T07:47:35Z

I think that the work is complete this feature.
Do you have any other comments?

jongyoul · 2016-06-13T08:50:13Z

LGTM. Merging if there's no more discussion.

cloverhearts · 2016-06-13T10:09:08Z

Thank you!

astroshim · 2016-06-15T06:42:21Z

LGTM

jongyoul · 2016-06-15T10:29:39Z

Merging it into master branch only.

### What is this PR for? Job management basic backend. was divided into smaller " PR ". - apache#921 Receive the basic data of Job manager api and the backend have been implemented. ### What type of PR is it? Feature ### Todos - [x] - Basic backend for job manager. - [x] - Basic backend api ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-963 ### How should this be tested? #### step 1. First, calling the rest api as follows receives the data of "job" of the whole. ```shell curl -H "Content-Type: application/json" -X GET http://127.0.0.1:8080/api/notebook/jobmanager/ ``` result ```json { "status":"OK", // result for api request "body":{ "lastResponseUnixTime":1465289938763, // last get server unixtime stamp. "jobs":[ // job list { "notebookId":"2BMQZ9QP6", // notebook id. "unixTimeLastRun":1465289017310, // notebook last running unixtime. "notebookType":"normal", // cron or normal "isRunningJob":false, // is Running? "notebookName":"Untitled Note 1226", // notebook name. "interpreter":"spark", // default interpreter group name or If you have not selected it does not exist. "paragraphs":[ { "name":"20160607-174331_232775609", // paragraph name 'undefined is notebook id' "id":"20160607-174331_232775609", // paragraph id "status":"FINISHED" // paragraph job status } ] } ] } } ``` #### step 2. For example, it showed the result of receiving the information as one of the notebook. focus on "lastResponseUnixTime" value. This value is inserted as an argument when you call the following restapi, us to be able to get the updated data. **Create a Notebook, or run the Paragraph.** And, **call updated notebook api** **"lastResponseUnixTime": 1465289938763** (step 1 get value) ```shell curl -H "Content-Type: application/json" -X GET http://127.0.0.1:8080/api/notebook/jobmanager/1465289938763 ``` #### step 3. If you created a Notebook, a new Notebook information is displayed. If there is a Notebook that if there is a Paragraph of the running, the information will also be displayed. #### step 4. When you restart the step1 of api, if you created a Notebook, the number of Notebook is +1 than when the first call. ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? yes Author: CloverHearts <cloverheartsdev@gmail.com> Closes apache#972 from cloverhearts/feat/sm/ZEPPELIN-531-basic-backend and squashes the following commits: 330ed73 [CloverHearts] change indent for jobmanager backend c7bf75a [CloverHearts] Jobmanager basic backend.

cloverhearts added 2 commits June 7, 2016 17:44

Jobmanager basic backend.

c7bf75a

change indent for jobmanager backend

330ed73

cloverhearts closed this Jun 7, 2016

cloverhearts reopened this Jun 7, 2016

cloverhearts closed this Jun 7, 2016

cloverhearts reopened this Jun 7, 2016

cloverhearts changed the title ~~ZEPPELIN-531 ] Jobmanagement - (1) basic backend.~~ ZEPPELIN-963 ] Jobmanagement - (1) basic backend. Jun 7, 2016

jongyoul reviewed Jun 9, 2016
View reviewed changes

asfgit closed this in cfe677b Jun 15, 2016

cloverhearts deleted the feat/sm/ZEPPELIN-531-basic-backend branch June 16, 2016 06:25

ZEPPELIN-963 ] Jobmanagement - (1) basic backend. #972

ZEPPELIN-963 ] Jobmanagement - (1) basic backend. #972

Uh oh!

Conversation

cloverhearts commented Jun 7, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is this PR for?

What type of PR is it?

Todos

What is the Jira issue?

How should this be tested?

step 1.

step 2.

step 3.

step 4.

Questions:

Uh oh!

cloverhearts commented Jun 7, 2016

Uh oh!

jongyoul commented Jun 7, 2016

Uh oh!

cloverhearts commented Jun 7, 2016

Uh oh!

jongyoul commented Jun 7, 2016

Uh oh!

cloverhearts commented Jun 7, 2016

Uh oh!

cloverhearts commented Jun 8, 2016

Uh oh!

astroshim commented Jun 8, 2016

Uh oh!

cloverhearts commented Jun 8, 2016

Uh oh!

cloverhearts commented Jun 9, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

astroshim commented Jun 9, 2016

Uh oh!

cloverhearts commented Jun 9, 2016

Uh oh!

jongyoul Jun 9, 2016

Choose a reason for hiding this comment

Uh oh!

cloverhearts Jun 10, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jongyoul Jun 10, 2016

Choose a reason for hiding this comment

Uh oh!

cloverhearts commented Jun 13, 2016

Uh oh!

jongyoul commented Jun 13, 2016

Uh oh!

cloverhearts commented Jun 13, 2016

Uh oh!

astroshim commented Jun 15, 2016

Uh oh!

jongyoul commented Jun 15, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cloverhearts commented Jun 7, 2016 •

edited

Loading

cloverhearts commented Jun 9, 2016 •

edited

Loading

cloverhearts Jun 10, 2016 •

edited

Loading