[SPARK-36237][UI][SQL] Attach and start handler after application started in UI #33457

AngersZhuuuu · 2021-07-21T06:35:23Z

What changes were proposed in this pull request?

When using prometheus to fetch metrics with a defined interval, we always pull data through restful API.
If the pulling happens when a driver SparkUI port is bind to the driver and the application is not fully started, Spark driver will throw a lot of exceptions about NoSuchElementException as below.

21/07/19 04:53:37 INFO Client: Preparing resources for our AM container
21/07/19 04:53:37 INFO Client: Uploading resource hdfs://tl3/packages/jars/spark-2.4-archive.tar.gz -> hdfs://R2/user/xiaoke.zhou/.sparkStaging/application_1624456325569_7143920/spark-2.4-archive.tar.gz
21/07/19 04:53:37 WARN JettyUtils: GET /jobs/ failed: java.util.NoSuchElementException: Failed to get the application information. If you are starting up Spark, please wait a while until it's ready.
java.util.NoSuchElementException: Failed to get the application information. If you are starting up Spark, please wait a while until it's ready.
	at org.apache.spark.status.AppStatusStore.applicationInfo(AppStatusStore.scala:43)
	at org.apache.spark.ui.jobs.AllJobsPage.render(AllJobsPage.scala:275)
	at org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:90)
	at org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:90)
	at org.apache.spark.ui.JettyUtils$$anon$3.doGet(JettyUtils.scala:90)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.spark_project.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
	at org.spark_project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
	at org.spark_project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
	at org.spark_project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513)
	at org.spark_project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
	at org.spark_project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.spark_project.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
	at org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
	at org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.spark_project.jetty.server.Server.handle(Server.java:539)
	at org.spark_project.jetty.server.HttpChannel.handle(Htt
[2021-07-19 04:54:55,111] INFO - pChannel.java:333)
	at org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
	at org.spark_project.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
	at org.spark_project.jetty.io.FillInterest.fillable(FillInterest.java:108)
	at org.spark_project.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
	at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
	at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
	at org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
	at java.lang.Thread.run(Thread.java:748)

Have check origin pr, we need to start server and bind port before taskScheduler started for client mode since we need to pass web url to register application master. But when we attach and start handler this time, we can provide restful API to user, but during this time, application is not started so we always return such error.

In this pr, to start SparUI, Spark starts Jetty Server first to bind address.
After the Spark application is fully started, call [attachAllHandlers] to start all existing handlers to Jetty seerver.

Why are the changes needed?

Improve the SparkUI start logical

Does this PR introduce any user-facing change?

Before spark application is fully started, all url request will return

Spark is starting up. Please wait a while until it's ready.

in the page

How was this patch tested?

Existed

During after bind address and finish start spark application, all request will show

…on started

AngersZhuuuu · 2021-07-21T06:35:52Z

ping @srowen since I have found you handle similar prs before in 2014

SparkQA · 2021-07-21T06:58:16Z

Test build #141391 has started for PR 33457 at commit 7832d40.

AngersZhuuuu · 2021-07-21T07:16:13Z

Related change
#1966
#2858

SparkQA · 2021-07-21T08:21:21Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45909/

SparkQA · 2021-07-21T09:01:53Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45909/

srowen

I don't know a lot about this, but it seems reasonable

SparkQA · 2021-07-21T11:58:17Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45923/

SparkQA · 2021-07-21T12:35:26Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45923/

SparkQA · 2021-07-21T13:45:45Z

Test build #141406 has finished for PR 33457 at commit 4fb422c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2021-07-21T15:48:13Z

ping @maropu @HyukjinKwon @cloud-fan

HyukjinKwon · 2021-07-22T02:08:36Z

cc @gengliangwang and @sarutak FYI

AngersZhuuuu · 2021-07-25T02:14:45Z

Any suggestion?

sarutak · 2021-07-26T05:48:45Z

@AngersZhuuuu
Before this change, the status code 500 is returned and helpful error message is shown if we access to /jobs before the UI is prepared.

But after this change, the status code 404 is returned and no helpful error message is shown.

It might be confusable for users.

AngersZhuuuu · 2021-07-26T06:42:58Z

It might be confusable for users.

With this 500 and error stack in the log makes user confused too.. they always ask me if there is something wong.
Always user won't open spark Web ui when not fully started .

Anyway expose a non-useable api is not reasonable? right?

AngersZhuuuu · 2021-07-26T07:07:38Z

Also you can check this related issue #1966

cloud-fan · 2021-07-26T11:35:39Z

Shall we make the RESTFUL request hang and the web page loading if the spark application is not fully started?

gengliangwang · 2021-07-26T15:46:32Z

When we use prometheus to fetch metrics, always pull data before application started.

we need to start server and bind port before taskScheduler started for client mode since we need to pass web url to register application master

@AngersZhuuuu could you update the PR description to make it more clear?

gengliangwang · 2021-07-26T15:50:03Z

With this 500 and error stack in the log makes user confused too.. they always ask me if there is something wong.

At least before the changes it shows hint "if you are starting spark..."

gengliangwang · 2021-07-26T15:59:18Z

Shall we make the RESTFUL request hang and the web page loading if the spark application is not fully started?

Or we can just redirect to a page saying Spark is starting

AngersZhuuuu · 2021-07-27T08:43:29Z

Shall we make the RESTFUL request hang and the web page loading if the spark application is not fully started?

Show as below is ok?

AngersZhuuuu · 2021-07-27T08:43:50Z

With this 500 and error stack in the log makes user confused too.. they always ask me if there is something wong.

At least before the changes it shows hint "if you are starting spark..."

How about current?

SparkQA · 2021-07-27T10:10:07Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46208/

AngersZhuuuu · 2021-07-27T10:26:07Z

retest this please

cloud-fan · 2021-07-27T10:48:29Z

#33457 (comment)

The web UI looks good. How about REST request?

SparkQA · 2021-07-28T06:44:36Z

Test build #141754 has finished for PR 33457 at commit b16531f.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-07-28T06:46:11Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46267/

SparkQA · 2021-07-28T07:33:18Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46268/

SparkQA · 2021-07-28T08:09:06Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46268/

SparkQA · 2021-07-28T09:38:37Z

Test build #141755 has finished for PR 33457 at commit b16531f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

core/src/main/scala/org/apache/spark/ui/SparkUI.scala

gengliangwang · 2021-07-28T13:58:13Z

@AngersZhuuuu Please update the PR description. Especially for the first and third ones.

AngersZhuuuu · 2021-07-28T14:36:38Z

@AngersZhuuuu Please update the PR description. Especially for the first and third ones.

How about current?

SparkQA · 2021-07-28T15:23:30Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46290/

SparkQA · 2021-07-28T15:35:47Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46291/

SparkQA · 2021-07-28T16:17:43Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46290/

SparkQA · 2021-07-28T17:12:27Z

Test build #141778 has finished for PR 33457 at commit c514855.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gengliangwang · 2021-07-29T13:33:58Z

core/src/test/scala/org/apache/spark/ui/UISuite.scala

+      .setMaster("local")
+      .setAppName("test")
+      .set(UI.UI_ENABLED, false)
+    val sc = new SparkContext(conf)


Is the test failure in GA related this the changes here?

Is the test failure in GA related this the changes here?

Seems so..

gengliangwang · 2021-07-29T13:38:25Z

@AngersZhuuuu seriously, the description is badly written.

When we use prometheus to fetch metrics, always pull data before application started.

Then throw a lot of exception not of NoSuchElementException

Also, please mention that there will be a hint saying that "Spark is starting up" in section "Does this PR introduce any user-facing change?"

AngersZhuuuu · 2021-07-29T17:36:05Z

@AngersZhuuuu seriously, the description is badly written.
When we use prometheus to fetch metrics, always pull data before application started.

Then throw a lot of exception not of NoSuchElementException
Also, please mention that there will be a hint saying that "Spark is starting up" in section "Does this PR introduce any user-facing change?"

How about current?

SparkQA · 2021-07-29T18:15:50Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46363/

SparkQA · 2021-07-29T20:06:51Z

Test build #141852 has finished for PR 33457 at commit 0c9c3ce.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2021-07-30T02:09:57Z

@gengliangwang GA passed now

AngersZhuuuu · 2021-08-02T07:53:28Z

ping @gengliangwang @sarutak

gengliangwang · 2021-08-02T11:35:43Z

Thanks, merging to master

gongzh021 · 2022-05-31T14:01:55Z

@AngersZhuuuu Before this change, the status code 500 is returned and helpful error message is shown if we access to /jobs before the UI is prepared.

But after this change, the status code 404 is returned and no helpful error message is shown.

It might be confusable for users.

I have the same bug.
Excuse me, how did you solve it?

AngersZhuuuu · 2022-05-31T14:10:39Z

@gongzh021 Maybe you can check this commit dba26cd

AngersZhuuuu · 2022-06-01T02:14:01Z

@gongzh021 Maybe you can check this commit dba26cd

[SPARK-36237][SQL] We should attach and start handler after applicati…

7832d40

…on started

github-actions bot added CORE WEB UI labels Jul 21, 2021

update

4fb422c

srowen reviewed Jul 21, 2021

View reviewed changes

HyukjinKwon changed the title ~~[SPARK-36237][SQL] We should attach and start handler after application started~~ [SPARK-36237][UI][SQL] Attach and start handler after application started in UI Jul 22, 2021

Update WebUI.scala

dba26cd

gengliangwang reviewed Jul 28, 2021

View reviewed changes

core/src/main/scala/org/apache/spark/ui/SparkUI.scala Outdated Show resolved Hide resolved

gengliangwang reviewed Jul 28, 2021

View reviewed changes

core/src/main/scala/org/apache/spark/ui/SparkUI.scala Outdated Show resolved Hide resolved

gengliangwang reviewed Jul 28, 2021

View reviewed changes

core/src/main/scala/org/apache/spark/ui/SparkUI.scala Outdated Show resolved Hide resolved

AngersZhuuuu added 2 commits July 28, 2021 22:34

Update SparkUI.scala

15aec1a

Update SparkUI.scala

c514855

gengliangwang reviewed Jul 29, 2021

View reviewed changes

Update UISuite.scala

0c9c3ce

gengliangwang approved these changes Aug 2, 2021

View reviewed changes

gengliangwang closed this in 951efb8 Aug 2, 2021

[SPARK-36237][UI][SQL] Attach and start handler after application started in UI #33457

[SPARK-36237][UI][SQL] Attach and start handler after application started in UI #33457

Uh oh!

Conversation

AngersZhuuuu commented Jul 21, 2021 • edited by gengliangwang Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

AngersZhuuuu commented Jul 21, 2021

Uh oh!

SparkQA commented Jul 21, 2021

Uh oh!

AngersZhuuuu commented Jul 21, 2021

Uh oh!

SparkQA commented Jul 21, 2021

Uh oh!

SparkQA commented Jul 21, 2021

Uh oh!

srowen left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 21, 2021

Uh oh!

SparkQA commented Jul 21, 2021

Uh oh!

SparkQA commented Jul 21, 2021

Uh oh!

AngersZhuuuu commented Jul 21, 2021

Uh oh!

HyukjinKwon commented Jul 22, 2021

Uh oh!

AngersZhuuuu commented Jul 25, 2021

Uh oh!

sarutak commented Jul 26, 2021

Uh oh!

AngersZhuuuu commented Jul 26, 2021

Uh oh!

AngersZhuuuu commented Jul 26, 2021

Uh oh!

cloud-fan commented Jul 26, 2021

Uh oh!

gengliangwang commented Jul 26, 2021

Uh oh!

gengliangwang commented Jul 26, 2021

Uh oh!

gengliangwang commented Jul 26, 2021

Uh oh!

AngersZhuuuu commented Jul 27, 2021

Uh oh!

AngersZhuuuu commented Jul 27, 2021

Uh oh!

SparkQA commented Jul 27, 2021

Uh oh!

AngersZhuuuu commented Jul 27, 2021

Uh oh!

cloud-fan commented Jul 27, 2021

Uh oh!

SparkQA commented Jul 28, 2021

Uh oh!

SparkQA commented Jul 28, 2021

Uh oh!

SparkQA commented Jul 28, 2021

Uh oh!

SparkQA commented Jul 28, 2021

Uh oh!

SparkQA commented Jul 28, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gengliangwang commented Jul 28, 2021

Uh oh!

AngersZhuuuu commented Jul 28, 2021

Uh oh!

SparkQA commented Jul 28, 2021

Uh oh!

SparkQA commented Jul 28, 2021

AngersZhuuuu commented Jul 21, 2021 •

edited by gengliangwang

Loading

AngersZhuuuu commented Aug 2, 2021 •

edited

Loading