[SPARK-5100][SQL]add webui for thriftserver #5730

tianyi · 2015-04-28T01:40:18Z

This PR is a rebased version of #3946 , and mainly focused on creating an independent tab for the thrift server in spark web UI.

Features:

Session related statistics ( username and IP are only supported in hive-0.13.1 )
List all the SQL executing or executed on this server
Provide links to the job generated by SQL
Provide link to show all SQL executing or executed in a specified session

Prototype snapshots:

This is the main page for thrift server

SparkQA · 2015-04-28T01:43:45Z

Test build #31097 has started for PR 5730 at commit 9830199.

liancheng · 2015-04-28T03:01:51Z

...e-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2.scala

totalExecution?

SparkQA · 2015-04-28T03:22:42Z

Test build #31097 has finished for PR 5730 at commit 9830199.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class HiveThriftServer2Listener(
This patch does not change any dependencies.

liancheng · 2015-04-28T03:24:40Z

...e-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2.scala

Append () to the method name since it causes side effect.

Same as above.

WangTaoTheTonic · 2015-04-28T03:58:09Z

Sorry for not following the previous comments but could you tell the difference between this tab and Thrift Server's driver UI?
Once this is added, will Thrift Server's Driver UI still be reserved?

liancheng · 2015-04-28T04:04:59Z

...e-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2.scala

Maybe onStatementCompiled? Also, executePlan should be executionPlan.

tianyi · 2015-04-28T04:59:23Z

@WangTaoTheTonic , the Thrift Server's driver UI you mentioned is a universal UI for all kinds of work on spark, and mainly focus on jobs, stages and tasks.
In this PR, we provide a new tab that focus on users, sessions and SQL.
So the driver UI won't be replaced since they are entirely different.

liancheng · 2015-04-28T05:11:38Z

...erver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerSessionPage.scala

Import order is wrong here.

liancheng · 2015-04-28T05:19:08Z

This generally looks good, left some comments, most are about styling issues.

@JoshRosen Would you mind to double check the web UI changes? This should be generally the same as #3946. Thanks! @tianyi will add Selenium based test cases later.

SparkQA · 2015-04-28T07:17:46Z

Test build #31128 has started for PR 5730 at commit c9df6f9.

tianyi · 2015-04-28T07:21:21Z

@JoshRosen I just add some tests for thrift server UI. But I'm not sure if I need to add test cases for all the feature. Any suggestion?

SparkQA · 2015-04-28T07:23:44Z

Test build #31129 has started for PR 5730 at commit 8809157.

SparkQA · 2015-04-28T07:39:44Z

Test build #31128 has finished for PR 5730 at commit c9df6f9.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

SparkQA · 2015-04-28T07:48:10Z

Test build #31129 has finished for PR 5730 at commit 8809157.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

SparkQA · 2015-04-28T08:08:54Z

Test build #31132 has started for PR 5730 at commit aa20408.

SparkQA · 2015-04-28T09:30:56Z

Test build #31132 has finished for PR 5730 at commit aa20408.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

AmplabJenkins · 2015-04-29T06:17:13Z

Merged build triggered.

AmplabJenkins · 2015-04-29T06:17:21Z

Merged build started.

SparkQA · 2015-04-29T06:17:58Z

Test build #31254 has started for PR 5730 at commit 0efe3d5.

SparkQA · 2015-04-29T06:42:20Z

Test build #31254 has finished for PR 5730 at commit 0efe3d5.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

AmplabJenkins · 2015-04-29T06:42:21Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-04-29T06:42:22Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31254/
Test FAILed.

AmplabJenkins · 2015-04-29T07:02:09Z

Merged build triggered.

AmplabJenkins · 2015-04-29T07:02:16Z

Merged build started.

SparkQA · 2015-04-29T07:02:48Z

Test build #31256 has started for PR 5730 at commit cfd14c7.

SparkQA · 2015-04-29T09:42:22Z

Test build #31256 has finished for PR 5730 at commit cfd14c7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

AmplabJenkins · 2015-04-29T09:42:27Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-04-29T09:42:28Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31256/
Test PASSed.

tianyi · 2015-04-29T09:52:27Z

@liancheng The test codes and is ready to review.

liancheng · 2015-05-04T08:56:43Z

LGTM, merging to master. Thanks for working on this!

marmbrus · 2015-05-06T18:32:14Z

The test added here is really noisy:

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/04/29 01:59:46 INFO UISeleniumSuite: Trying to start HiveThriftServer2: port=19866, mode=binary, attempt=0
15/04/29 02:00:16 INFO UISeleniumSuite: HiveThriftServer2 started successfully
15/04/29 02:00:19 WARN DefaultCssErrorHandler: CSS error: 'http://localhost:24602/static/bootstrap.min.css' [10:11] Error in style rule. (Invalid token "*". Was expecting one of: <EOF>, <S>, <IDENT>, "}", ";".)
15/04/29 02:00:19 WARN DefaultCssErrorHandler: CSS warning: 'http://localhost:24602/static/bootstrap.min.css' [10:11] Ignoring the following declarations in this rule.
15/04/29 02:00:19 WARN DefaultCssErrorHandler: CSS error: 'http://localhost:24602/static/bootstrap.min.css' [15:41] Error in style rule. (Invalid token "*". Was expecting one of: <EOF>, <S>, <IDENT>, "}", ";".)
15/04/29 02:00:19 WARN DefaultCssErrorHandler: CSS warning: 'http://localhost:24602/static/bootstrap.min.css' [15:41] Ignoring the following declarations in this rule.
15/04/29 02:00:19 WARN DefaultCssErrorHandler: CSS error: 'http://localhost:24602/static/bootstrap.min.css' [26:14] Error in style rule. (Invalid token "*". Was expecting one of: <EOF>, <S>, <IDENT>, "}", ";".)
15/04/29 02:00:19 WARN DefaultCssErrorHandler: CSS warning: 'http://localhost:24602/static/bootstrap.min.css' [26:14] Ignoring the following declarations in this rule.
15/04/29 02:00:19 WARN DefaultCssErrorHandler: CSS error: 'http://localhost:24602/static/bootstrap.min.css' [39:24] Error in style rule. (Invalid token "*". Was expecting one of: <EOF>, <S>, <IDENT>, "}", ";".)

Can you submit a followup to quiet it down?

tianyi · 2015-05-07T03:39:22Z

OK, I will work on it.

tianyi · 2015-05-08T08:02:41Z

@marmbrus the test issue is fixed in #5983

This PR is a rebased version of apache#3946 , and mainly focused on creating an independent tab for the thrift server in spark web UI. Features: 1. Session related statistics ( username and IP are only supported in hive-0.13.1 ) 2. List all the SQL executing or executed on this server 3. Provide links to the job generated by SQL 4. Provide link to show all SQL executing or executed in a specified session Prototype snapshots: This is the main page for thrift server ![image](https://cloud.githubusercontent.com/assets/1411869/7361379/df7dcc64-ed89-11e4-9964-4df0b32f475e.png) Author: tianyi <tianyi.asiainfo@gmail.com> Closes apache#5730 from tianyi/SPARK-5100 and squashes the following commits: cfd14c7 [tianyi] style fix 0efe3d5 [tianyi] revert part of pom change c0f2fa0 [tianyi] extends HiveThriftJdbcTest to start/stop thriftserver for UI test aa20408 [tianyi] fix style problem c9df6f9 [tianyi] add testsuite for thriftserver ui and fix some style issue 9830199 [tianyi] add webui for thriftserver

add webui for thriftserver

9830199

tianyi changed the title ~~add webui for thriftserver~~ [SPARK-5100][SQL]add webui for thriftserver Apr 28, 2015

tianyi mentioned this pull request Apr 28, 2015

[SPARK-5100][SQL] add thriftserver-ui support #3946

Closed

liancheng reviewed Apr 28, 2015
View reviewed changes

...e-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2.scala Outdated

Copy link

Contributor

liancheng Apr 28, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

totalExecution?

liancheng reviewed Apr 28, 2015
View reviewed changes

add testsuite for thriftserver ui and fix some style issue

c9df6f9

fix style problem

aa20408

tianyi force-pushed the SPARK-5100 branch from 8809157 to aa20408 Compare April 28, 2015 08:04

tianyi added 2 commits April 29, 2015 14:12

extends HiveThriftJdbcTest to start/stop thriftserver for UI test

c0f2fa0

revert part of pom change

0efe3d5

style fix

cfd14c7

asfgit closed this in 343d3bf May 4, 2015

[SPARK-5100][SQL]add webui for thriftserver #5730

[SPARK-5100][SQL]add webui for thriftserver #5730

Uh oh!

Conversation

tianyi commented Apr 28, 2015

Uh oh!

SparkQA commented Apr 28, 2015

Uh oh!

liancheng Apr 28, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 28, 2015

Uh oh!

liancheng Apr 28, 2015

Choose a reason for hiding this comment

Uh oh!

liancheng Apr 28, 2015

Choose a reason for hiding this comment

Uh oh!

WangTaoTheTonic commented Apr 28, 2015

Uh oh!

liancheng Apr 28, 2015

Choose a reason for hiding this comment

Uh oh!

tianyi commented Apr 28, 2015

Uh oh!

liancheng Apr 28, 2015

Choose a reason for hiding this comment

Uh oh!

liancheng commented Apr 28, 2015

Uh oh!

SparkQA commented Apr 28, 2015

Uh oh!

tianyi commented Apr 28, 2015

Uh oh!

SparkQA commented Apr 28, 2015

Uh oh!

SparkQA commented Apr 28, 2015

Uh oh!

SparkQA commented Apr 28, 2015

Uh oh!

SparkQA commented Apr 28, 2015

Uh oh!

SparkQA commented Apr 28, 2015

Uh oh!

AmplabJenkins commented Apr 29, 2015

Uh oh!

AmplabJenkins commented Apr 29, 2015

Uh oh!

SparkQA commented Apr 29, 2015

Uh oh!

SparkQA commented Apr 29, 2015

Uh oh!

AmplabJenkins commented Apr 29, 2015

Uh oh!

AmplabJenkins commented Apr 29, 2015

Uh oh!

AmplabJenkins commented Apr 29, 2015

Uh oh!

AmplabJenkins commented Apr 29, 2015

Uh oh!

SparkQA commented Apr 29, 2015

Uh oh!

SparkQA commented Apr 29, 2015

Uh oh!

AmplabJenkins commented Apr 29, 2015

Uh oh!

AmplabJenkins commented Apr 29, 2015

Uh oh!

tianyi commented Apr 29, 2015

Uh oh!

liancheng commented May 4, 2015

Uh oh!

marmbrus commented May 6, 2015

Uh oh!

tianyi commented May 7, 2015

Uh oh!

tianyi commented May 8, 2015

Uh oh!

Reviewers