ZEPPELIN-1692: Ability to access Spark jobs UI from the paragraph #1663

karuppayya · 2016-11-21T15:32:34Z

What is this PR for?

A paragraph execution may result in spark job(s).
Adding ability to access the spark job UI(corresponding to the job generated by the paragraph run), directly from the paragraph.

What type of PR is it?

Improvement

Todos

Write tests

What is the Jira issue?

ZEPPELIN-1692

How should this be tested?

Run paragraphs with spark code(scala, pyspark, sql, R).
The paragraph will display a button on the top right corner, which on click will open up the corresponding job UI

Screenshots (if appropriate)

Questions:

Does the licenses files need update? NA
Is there breaking changes for older versions? NA
Does this needs documentation? NA

cloverhearts · 2016-11-21T15:58:57Z

awsome!

karuppayya · 2016-11-21T16:11:22Z

zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Paragraph.java

  Date dateUpdated;
  private Map<String, Object> config; // paragraph configs like isOpen, colWidth, etc
  public final GUI settings;          // form and parameter settings
+  private Map<String, Set<String>> runtimeInfos;


I dont want to persist this info in json.
When i add the transient modifier, the serailization to JSON(for broadcasting to client browser) doesnot happen.
I can add an Exclusion strategy for all the storage handlers. But wanted to check if there is an easier way to achieve this . @Leemoonsoo @cloverhearts

Hello,
I think there is no problem using transient.
If you think you have a problem with using transient, can you comment on reason?

Actually, my english is not good.
I am sorry if I have been rude to you.
Thank you.

@cloverhearts Your English is not bad 👍

If i add transient modifier, when Gson.toJson(Paragraph) is done to send message about note/paragraph, the runtimeInfos wont be serialized and will not be received by browser(runtimeInfos wont get persisted in note.json),

Whereas if i add the modifier(current implementation), it will get persisted in JSON. But we dont want it, since it a run time information, and we might not want to persist in the note.json

Is there a way where i can serailize this info(for sending to client browser) as well as not persist in json?
PS: runtimeInfos object has the paragraph url information.

karuppayya · 2016-11-23T01:54:17Z

Ready for review @Leemoonsoo

1ambda · 2016-11-23T05:42:02Z

Let me review this PR and then give you some feedbacks. Thanks @karup1990!

I think Spark job button might be located on the left of the FINISHED button to keep consistency with other interpreter results which are not using spark. Maybe we can put it on the top left corner of a paragraph and other interpreters can use the location to give their interpreter specific buttons, options.
What about using uppercase chars like SPARK JOB because we are already using uppercases in FINISHED
If when I clicked hide output button on a paragraph, Spark job button doesn't disappear. What do you think? Should it be removed?

karuppayya · 2016-11-23T18:50:24Z

Thanks @1ambda for looking into this PR and for the suggestions.
For the 3rd point,
I am not very sure of the use cases for hiding the output in a paragraph.
Do you see a case where the job link has to be hidden when output is hidden?

1ambda · 2016-11-23T22:42:26Z

@karup1990 sorry for misspelling It was clear output

zjffdu · 2016-11-23T23:38:46Z

spark/src/main/java/org/apache/zeppelin/spark/ZeppelinContext.java

    Method take;
-    String jobGroup = "zeppelin-" + interpreterContext.getParagraphId();
+    String jobGroup = "zeppelin-" + interpreterContext.getNoteId() + "-"
+        + interpreterContext.getParagraphId();


I see several places of constructing jobGroupId from noteId/paragraphId and extract noteId/paragraphId from jobGroupId. It's better to put them together so that others can understand it easily and other places can reuse it.

I have added Utils. buildJobGroupId and using it across.

zjffdu · 2016-11-23T23:41:31Z

...reter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterEventPoller.java

+    String id = interpreterGroup.getId();
+    int indexOfColon = id.indexOf(":");
+    return id.substring(0, indexOfColon);
+  }


This assume the interpreterGroup id format, seems fragile.

I am seeing that interpreter setting is the first prefix, then based on the interpreter mode we are appending ids. Please correct me if I am wrong .

I think it is better to put getInterpreterSettingId into Utils as well because it is about how we parse interpreterGroup

zjffdu · 2016-11-24T04:40:58Z

spark/src/main/java/org/apache/zeppelin/spark/SparkRInterpreter.java

+    SparkInterpreter sparkInterpreter = getSparkInterpreter();
+    sparkInterpreter.populateSparkWebUrl(interpreterContext);
+
+    String jobGroup = sparkInterpreter.getJobGroup(interpreterContext);


Would it be better to rename it sparkInterpreter.buildJobGroupId as each time it would create a new jobGroupId IIUC.

zjffdu · 2016-11-24T04:44:38Z

I left some comments inline and could you add test for this feature ?

AhyoungRyu · 2016-11-24T06:01:41Z

zeppelin-web/src/app/notebook/paragraph/paragraph-control.html

+    <a href="{{paragraph.runtimeInfos.jobUrl[0]}}" target="_blank"><span class="fa fa-tasks"></span> Spark job </a>
+  </span>
+  <span class="dropdown" ng-show="paragraph.runtimeInfos.jobUrl.length > 1">
+    <span class="fa fa-tasks" style="cursor:pointer;color:#3071A9" tooltip-placement="top" tooltip="Run this paragraph (Shift+Enter)"


I think tooltip-placement="top" tooltip="Run this paragraph (Shift+Enter)" doesn't need in here :)

AhyoungRyu · 2016-11-24T06:22:48Z

Hi @karup1990, really nice and useful feature indeed.
While quickly looking through this, I have some minor feedbacks about UI :)

Spark Job and Spark jobs have different font like below.
And I think these block that you added in this PR should be below this line.

  <span ng-show="paragraph.runtimeInfos.jobUrl.length == 1">
    <a href="{{paragraph.runtimeInfos.jobUrl[0]}}" target="_blank"><span class="fa fa-tasks"></span> Spark Job </a>
  </span>
  <span class="dropdown" ng-show="paragraph.runtimeInfos.jobUrl.length > 1">
  <span class="fa fa-tasks" style="cursor:pointer;color:#3071A9" tooltip-placement="top" tooltip="Run this paragraph (Shift+Enter)"
        data-toggle="dropdown"
        type="button">  Spark Jobs
  </span>
    <ul class="dropdown-menu" role="menu" style="width:200px;z-index:1002">
       <li ng-class="{'option-disabled': !noteOperationsAllowed()}" ng-repeat="url in paragraph.runtimeInfos.jobUrl">
         <a href="{{url}}" target="_blank"><span class="fa fa-external-link-square"></span> Jobs #{{$index}}</a>
       </li>
    </ul>
  </span>

current
after

AhyoungRyu · 2016-11-24T06:07:39Z

zeppelin-web/src/app/notebook/paragraph/paragraph-control.html

        ng-click="runParagraph(getEditorValue())"
        ng-show="paragraph.status!='RUNNING' && paragraph.status!='PENDING' && paragraph.config.enabled"></span>
+  <span ng-show="paragraph.runtimeInfos.jobUrl.length == 1">
+    <a href="{{paragraph.runtimeInfos.jobUrl[0]}}" target="_blank"><span class="fa fa-tasks"></span> Spark job </a>


Maybe it's nit pick, let's use capital letter for Spark job as below "Spark Jobs" does :)

How about display runtime info button before paragraph status and run button?

So all other other control buttons are stay in the same position whether runtimeInfo button is displayed or not.

And how about we generalize this feature little bit more, such as

runtimeInfos = { label: "Spark Job" jobUrl : [], }

So other interpreters can also leverage this feature.

@1ambda Clearing the links when we clear the output 👍
@AhyoungRyu Fixed font issue
@Leemoonsoo Added ParagraphRuntimeInfos class(https://github.com/apache/zeppelin/pull/1663/files#diff-a2ac97dfb3f07b2a3fd1533af701295e) to have a generic support to add more properties. Let me know how it looks.
@AhyoungRyu @Leemoonsoo @1ambda Fixed the alignment and capitalized the text(to access the jobs)

karuppayya · 2016-11-25T16:31:41Z

Thanks for the feedback. i will have them handled some time over the weekend.

karuppayya · 2016-11-26T04:29:26Z

Failure seems not related
04:09:12,112 ERROR org.apache.zeppelin.AbstractZeppelinIT:136 - Exception in ZeppelinIT while testAngularDisplay org.openqa.selenium.TimeoutException: Timed out after 60 seconds waiting for org.apache.zeppelin.AbstractZeppelinIT$1@4c7305e4 Build info: version: '2.48.2', revision: '41bccdd10cf2c0560f637404c2d96164b67d9d67', time: '2015-10-09 13:08:06' System info: host: 'testing-docker-bcffa7c6-8593-4893-a72f-07a719f105fa', ip: '172.17.0.6', os.name: 'Linux', os.arch: 'amd64', os.version: '4.8.7-040807-generic', java.version: '1.7.0_76' Driver info: driver.version: unknown

karuppayya · 2016-11-27T13:54:50Z

Updated the PR with test @zjffdu

karuppayya · 2016-11-30T15:42:23Z

Ready for review

zjffdu · 2016-12-01T00:59:02Z

spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java

+        int secondIndex = jobgroupId.indexOf("-", indexOf + 1);
+        return jobgroupId.substring(secondIndex + 1, jobgroupId.length());
+      }
+    };


Would it be better to put getNoteId and getParagraphId to Utils ? So that the constructing and parsing logic are in the same class.

zjffdu · 2016-12-01T01:03:20Z

zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/ParagraphRuntimeInfos.java

+
+  public void addValue(String value) {
+    if (values == null) {
+      values = new ArrayList<>();


Should we create values in the constructor ? Otherwise at least this method is not thread-safe (we might create 2 values in 2 threads)

zjffdu · 2016-12-01T01:07:47Z

zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Paragraph.java

  Date dateUpdated;
  private Map<String, Object> config; // paragraph configs like isOpen, colWidth, etc
  public final GUI settings;          // form and parameter settings
+  private Map<String, ParagraphRuntimeInfos> runtimeInfos;


Should we rename ParagraphRuntimeInfos to ParagraphRuntimeInfo ? Because it is a little confusing that runtimeInfos is also plural

zjffdu · 2016-12-01T01:09:55Z

zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java

      if (p.getDateFinished() != null && lastUpdatedDate.before(p.getDateFinished())) {
        lastUpdatedDate = p.getDateFinished();
      }
+      p.clearRuntimeInfo();


Why we need to clear RuntimeInfo when we load note if the runtimeInfo is not stored ?

I will nedd to remove this line.
The runtime infos is persisted currently. But I dont want to persist it.
If i add transient modifier to runtimeInfos, when Gson.toJson(Paragraph) is done to send message about note/paragraph, the runtimeInfos wont be serialized and will not be received by browser(runtimeInfos wont get persisted in note.json),

Whereas if i add the modifier(current implementation), it will get persisted in JSON. But we dont want it, since it a run time information, and we might not want to persist in the note.json

Is there a way where i can serialize this info(for sending to client browser) as well as not persist in json?
PS: runtimeInfos object has the paragraph url information.

zjffdu · 2016-12-01T01:22:38Z

zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/ParagraphRuntimeInfos.java

+
+  String propertyName;
+  String label;
+  String group;


I think here the label and group are duplicated. I don't see you use label in frontend. Another approach I think of is that we always use interpreter group name as the prefix of propertyName as convention, (e.g. spark.jobUrl), so that we don't need label or group. @Leemoonsoo

The label is used here

zeppelin/zeppelin-web/src/app/notebook/paragraph/paragraph-control.html

Line 19 in 9612dd0

{{paragraph.runtimeInfos.jobUrl.label}}

I had added group , so that two interpreters having same property name can be rendered in a different manner in UI.

zjffdu · 2016-12-01T01:28:14Z

spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java

      Map<String, String> infos = new java.util.HashMap<>();
      if (sparkUrl != null) {
        infos.put("url", sparkUrl);
        logger.info("Sending metainfos to Zeppelin server: {}", infos.toString());


Should we put the logging in the if block ? And what about else block ? Will that happen ?

zjffdu · 2016-12-01T01:49:41Z

zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java

+    }
+  }
+
+  public void clearParagraphRuntimeInfo(InterpreterSetting setting) {


I am afraid this method may some issues because the binding between InterpreterSetting and Note/Paragraph may change. e.g. At the beginning I use spark in p1, but later I use flink in p1, but the method will clear all the runtime info of p1 when I restart spark interpreter.

Right.. I have handled this now.

Signed-off-by: karuppayya <karuppayya1990@gmail.com>

Signed-off-by: Karup <karuppayya@outlook.com>

Signed-off-by: karuppayya <karuppayya1990@gmail.com>

Signed-off-by: karuppayya <karuppayyar@qubole.com>

Signed-off-by: Karup <karuppayya@outlook.com>

karuppayya · 2017-02-04T01:30:30Z

Thanks @Leemoonsoo for the suggestion and the pointer.
Looks like all tested have passed. Ready to merge.

Leemoonsoo · 2017-02-04T02:51:12Z

@karuppayya Great work!

LGTM and merge to master if no further discussions.

Tagar · 2017-03-01T21:33:37Z

Just upgraded to yesterday's master snapshot.

When I click on any of these links, link is leading to

http://hostname.domain.com:8088/proxy/application_1488384993892_0001/jobs/job/

and that error shows HTTP ERROR 400

Page reads

Problem accessing /jobs/job/.

Reason: requirement failed: Missing id parameter
Powered by Jetty://

Thanks.

karuppayya · 2017-03-03T12:56:00Z

@Tagar to access a specific job, the id has to be part of the url .
Something like http://hostname.domain.com:8088/proxy/application_1488384993892_0001/jobs/job?id=
I am not sure why the id is not part of the URL. Did you find anything suspicious in logs? Also were you able to access the jobs from Spark Web UI?
Is it consistent?

mindcrusher11 · 2017-03-03T13:58:26Z

not sure but http://sparkkafka:4041/streaming/batch/?id=1488549398000 from history server job ids can be accessed.

…

On Fri, Mar 3, 2017 at 12:56 PM, Karuppayya ***@***.***> wrote: [image: Boxbe] <https://www.boxbe.com/overview> This message is eligible for Automatic Cleanup! ***@***.***) Add cleanup rule <https://www.boxbe.com/popup?url=https%3A%2F%2Fwww.boxbe.com%2Fcleanup%3Fkey%3DhZ2jaDHPZTj0KuKoaNm7fK9e%252FzMLHTLqTv%252FFNfodeFk%253D%26token%3DaEdD%252F%252BeLmCv0tCOGsZpb4o%252BEK5LEtxCCSSsv8PPya3JSu%252F5lN7vTnmpP2Sg9oTH7ezUWL7vqbNgezPE0xVQ39d%252BN5cY1norzDv3iDnMds%252FGzIQhRKIfX4iLHJowWwPdmejdhDctIKx2WZ8%252F%252FVC6yVQ%253D%253D&tc_serial=29148464902&tc_rand=1062463868&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001> | More info <http://blog.boxbe.com/general/boxbe-automatic-cleanup?tc_serial=29148464902&tc_rand=1062463868&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001> @Tagar <https://github.com/Tagar> to access a specific job, the id has to be part of the url . Something like http://hostname.domain.com:8088/proxy/application_ 1488384993892_0001/jobs/job?id= I am not sure why the id is not part of the URL. Did you find anything suspicious in logs? Also were you able to access the jobs from Spark Web UI? Is it consistent? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1663 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACVt9jetc90A_mWk4WINCAQ4V3ekg8eMks5riA3igaJpZM4K4Vff> .

Tagar · 2017-03-03T17:51:23Z

@karuppayya thank you for the follow up.

A little bit more information - the link on paragprah leads to

http://10.20.32.57:28009/jobs/job?id=123

A Spark Driver UI.
Spark Driver web server then redirects to a link like I sent in my previous post,
and it misses job id in the redirected URL, i.e.

http://host.domain.com:8088/proxy/application_1488384993892_0044/jobs/job/

To your question on consistency - yes it happens in 100% cases, I never saw this new feature works for us.
Is this Spark version dependent or something? We're running CDH Spark 2.

Tagar · 2017-03-06T19:50:12Z

According to apache/spark#5947 URL format is different in YARN and non-YARN modes? Was PR-1663 for ZEPPELIN-1692 tested on both of these modes? Not sure what else might break those links. We do use Spark on YARN. Created https://issues.apache.org/jira/browse/ZEPPELIN-2221 to move discussion there. Thank you.

A paragraph execution may result in spark job(s). Adding ability to access the spark job UI(corresponding to the job generated by the paragraph run), directly from the paragraph. Improvement * [x] Write tests ZEPPELIN-1692 Run paragraphs with spark code(scala, pyspark, sql, R). The paragraph will display a button on the top right corner, which on click will open up the corresponding job UI ![spark_jobs](https://cloud.githubusercontent.com/assets/5082742/20488337/a07fe35a-b02c-11e6-9400-db9f1c10df90.gif) * Does the licenses files need update? NA * Is there breaking changes for older versions? NA * Does this needs documentation? NA Author: Karup <karuppayya@outlook.com> Author: karuppayya <karuppayya1990@gmail.com> Author: karuppayya <karuppayyar@qubole.com> Closes apache#1663 from karuppayya/ZEPPELIN-1692 and squashes the following commits: 4253d0b [Karup] Fix bad rebase d7eb3b6 [Karup] Fix paragraph.js 8e2cd85 [Karup] tryout: fix selenium tests based on moons suggstion 732b0a4 [karuppayya] Fix test 890107d [Karup] Fix test - tryout ed4685c [Karup] Fix tooltip d27221d [Karup] Adding license header 87214a7 [Karup] Fix incorrect rebase 19513a6 [Karup] Send para runtimeinfos via websocker, but dont persist in json 09fc0e2 [Karup] Fix compilation fc44d9b [Karup] Address review comments b837c6c [karuppayya] Fix incorrect variable used 42d92ac [karuppayya] Fix test d4e54e8 [karuppayya] Address review feedbacks 1a45284 [Karup] Fix test 717eedf [Karup] Add tests , refactor 25379aa [Karup] Clear job urls when we clear output 7383c0a [Karup] Address review comments e2cd4db [karuppayya] Fix NPE in tests 3d9a573 [karuppayya] Fix NPE and some refactoring 9b3a3e2 [karuppayya] Fix checkstyle f16422f [karuppayya] Ability to view spark job urls in each paragraph (cherry picked from commit e9caebc)

Tagar · 2017-05-16T18:34:19Z

Does this feature work for anyone who is using Spark on YARN?

It seems to be broken by https://issues.apache.org/jira/browse/SPARK-20772

I've updated https://issues.apache.org/jira/browse/ZEPPELIN-2221

karuppayya commented Nov 21, 2016

View reviewed changes

karuppayya force-pushed the ZEPPELIN-1692 branch from 2fc4dcf to ad50955 Compare November 22, 2016 15:58

zjffdu reviewed Nov 23, 2016

View reviewed changes

zjffdu reviewed Nov 24, 2016

View reviewed changes

AhyoungRyu reviewed Nov 24, 2016

View reviewed changes

karuppayya closed this Nov 26, 2016

karuppayya reopened this Nov 26, 2016

karuppayya force-pushed the ZEPPELIN-1692 branch from 4fcc24e to d2d633e Compare November 26, 2016 04:30

karuppayya force-pushed the ZEPPELIN-1692 branch from e8fd1e0 to 9612dd0 Compare November 30, 2016 11:58

zjffdu reviewed Dec 1, 2016

View reviewed changes

karuppayya force-pushed the ZEPPELIN-1692 branch from f85bfcc to d978c48 Compare December 1, 2016 16:21

karuppayya and others added 17 commits February 2, 2017 13:42

Fix NPE in tests

e2cd4db

Signed-off-by: karuppayya <karuppayya1990@gmail.com>

Address review comments

7383c0a

Signed-off-by: Karup <karuppayya@outlook.com>

Clear job urls when we clear output

25379aa

Signed-off-by: Karup <karuppayya@outlook.com>

Add tests , refactor

717eedf

Signed-off-by: Karup <karuppayya@outlook.com>

Fix test

1a45284

Signed-off-by: Karup <karuppayya@outlook.com>

Address review feedbacks

d4e54e8

Signed-off-by: karuppayya <karuppayya1990@gmail.com>

Fix test

42d92ac

Signed-off-by: karuppayya <karuppayya1990@gmail.com>

Fix incorrect variable used

b837c6c

Signed-off-by: karuppayya <karuppayyar@qubole.com>

Address review comments

fc44d9b

Signed-off-by: Karup <karuppayya@outlook.com>

Fix compilation

09fc0e2

Signed-off-by: Karup <karuppayya@outlook.com>

Send para runtimeinfos via websocker, but dont persist in json

19513a6

Fix incorrect rebase

87214a7

Adding license header

d27221d

Fix tooltip

ed4685c

Fix test - tryout

890107d

Fix test

732b0a4

tryout: fix selenium tests based on moons suggstion

8e2cd85

karuppayya force-pushed the ZEPPELIN-1692 branch from 27ca32c to 8e2cd85 Compare February 2, 2017 08:15

Karup added 2 commits February 3, 2017 17:58

Fix paragraph.js

d7eb3b6

Fix bad rebase

4253d0b

asfgit closed this in e9caebc Feb 5, 2017

ZEPPELIN-1692: Ability to access Spark jobs UI from the paragraph #1663

ZEPPELIN-1692: Ability to access Spark jobs UI from the paragraph #1663

Uh oh!

Conversation

karuppayya commented Nov 21, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is this PR for?

What type of PR is it?

Todos

What is the Jira issue?

How should this be tested?

Screenshots (if appropriate)

Questions:

Uh oh!

cloverhearts commented Nov 21, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

karuppayya commented Nov 23, 2016

Uh oh!

1ambda commented Nov 23, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

karuppayya commented Nov 23, 2016

Uh oh!

1ambda commented Nov 23, 2016

Uh oh!

zjffdu Nov 23, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

karuppayya Nov 26, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zjffdu commented Nov 24, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AhyoungRyu commented Nov 24, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

karuppayya commented Nov 25, 2016

Uh oh!

karuppayya commented Nov 26, 2016

Uh oh!

karuppayya commented Nov 27, 2016

Uh oh!

karuppayya commented Nov 30, 2016

Uh oh!

zjffdu Dec 1, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zjffdu Dec 1, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

karuppayya commented Nov 21, 2016 •

edited

Loading

1ambda commented Nov 23, 2016 •

edited

Loading

zjffdu Nov 23, 2016 •

edited

Loading

karuppayya Nov 26, 2016 •

edited

Loading

zjffdu Dec 1, 2016 •

edited

Loading

zjffdu Dec 1, 2016 •

edited

Loading

zjffdu Dec 1, 2016 •

edited

Loading