Fix serverless job run failed related issues #2899

t-rufang · 2019-03-21T17:21:32Z

With this PR,

Serverless job can successfully run and finish when backend is stable.
Pop up Spark master UI web page when serverless job is starting
Warn user that kill serverless job action is invalid when the job is in Finalizing or Ended state

wezhang · 2019-03-22T02:15:39Z

...ode-common/src/com/microsoft/azure/hdinsight/spark/common/CosmosServerlessSparkBatchJob.java

+
+    public boolean isJobEnded(
+            @NotNull com.microsoft.azure.hdinsight.sdk.rest.azure.serverless.spark.models.SparkBatchJob sparkBatchJob) {
+        return sparkBatchJob.schedulerState().toString().equalsIgnoreCase(SchedulerState.ENDED.toString());


Do you need adding some abnormal end status, such as: failed, killed? #Resolved

wezhang · 2019-03-22T02:18:55Z

...ode-common/src/com/microsoft/azure/hdinsight/spark/common/CosmosServerlessSparkBatchJob.java

-                .onErrorResumeNext(err -> {
-                    if (err instanceof SparkJobFinishedException || err.getCause() instanceof SparkJobFinishedException) {
-                        return Observable.error(err);
+                .takeUntil(sparkBatchJob ->


The val sparkBatchJob might be easy confused. You can try batchResponse or batchResp #Resolved

wezhang · 2019-03-22T02:19:41Z

...ode-common/src/com/microsoft/azure/hdinsight/spark/common/CosmosServerlessSparkBatchJob.java

+        final int GET_JOB_DONE_REPEAT_DELAY_MILLISECONDS = 1000;
+        return getSparkBatchJobRequest()
+                .repeatWhen(ob -> ob.delay(GET_JOB_DONE_REPEAT_DELAY_MILLISECONDS, TimeUnit.MILLISECONDS))
+                .takeUntil(sparkBatchJob -> {


The same, batchResp or others #Resolved

wezhang · 2019-03-22T02:21:18Z

...ode-common/src/com/microsoft/azure/hdinsight/spark/common/CosmosServerlessSparkBatchJob.java

+                .takeUntil(sparkBatchJob -> {
+                    String jobState = getJobState(sparkBatchJob);
+                    return isJobEnded(sparkBatchJob)
+                            || (jobState != null && isDone(jobState));


What's the difference between isDone and isJobEnded? Can they be merged? #Resolved

wezhang · 2019-03-22T02:22:57Z

...ode-common/src/com/microsoft/azure/hdinsight/spark/common/CosmosServerlessSparkBatchJob.java

+        ctrlInfo("The Spark job is starting...");
+        return getSparkBatchJobRequest()
+                .doOnNext(sparkBatchJob -> {
+                    if (sparkBatchJob.properties() != null && sparkBatchJob.properties().sparkMasterUI() != null) {


The sparkBatchJob.properties() != null && sparkBatchJob.properties().sparkMasterUI() can be wrapped as a getMasterUI(batchResp) method #Resolved

wezhang · 2019-03-22T02:24:39Z

...ode-common/src/com/microsoft/azure/hdinsight/spark/common/CosmosServerlessSparkBatchJob.java

+                                .takeUntil(sparkBatchJob -> {
+                                    String jobState = getJobState(sparkBatchJob);
+                                    return isJobEnded(sparkBatchJob)
+                                            || (jobState != null && (isDone(jobState) || isRunning(jobState)));


If isJobEnded and isDone merged, the code can looks like: isJobEnded() || isJobStarted() #Resolved

Yes, these two methods are confusing, I will merge them then.

In reply to: 268020311 [](ancestors = 268020311)

wezhang · 2019-03-22T02:25:18Z

...ode-common/src/com/microsoft/azure/hdinsight/spark/common/CosmosServerlessSparkBatchJob.java

+                })
+                .flatMap(job ->
+                        getSparkBatchJobRequest()
+                                .repeatWhen(ob -> ob.delay(getDelaySeconds(), TimeUnit.SECONDS))


You can keep "The Spark job is starting..." output in repeat waiting function. #Resolved

wezhang · 2019-03-22T02:26:31Z

...ode-common/src/com/microsoft/azure/hdinsight/spark/common/CosmosServerlessSparkBatchJob.java

+                                .flatMap(sparkBatchJob -> {
+                                    String jobState = getJobState(sparkBatchJob);
+                                    if (jobState != null && (isDone(jobState) || isRunning(jobState))) {
+                                        if (isDone(jobState) && !isSuccess(jobState)) {


Revert the if..else might be more clear and easy understanding. #Resolved

wezhang · 2019-03-22T02:27:36Z

...ode-common/src/com/microsoft/azure/hdinsight/spark/common/CosmosServerlessSparkBatchJob.java

+                .takeUntil(sparkBatchJob ->
+                        isJobEnded(sparkBatchJob)
+                                || (sparkBatchJob.properties() != null
+                                && StringUtils.isNotEmpty(sparkBatchJob.properties().livyServerAPI())))


The same as above, getLivyAPI() #Resolved

wezhang · 2019-03-22T02:32:38Z

...ode-common/src/com/microsoft/azure/hdinsight/spark/common/CosmosServerlessSparkBatchJob.java

+                                                        String message = "Job is in " + logAndStateTriple.getMiddle() + " state.";
+                                                        return Observable.just(new AbstractMap.SimpleImmutableEntry<>(Log, message));
+                                                    } else if (logAndStateTriple.getRight().equalsIgnoreCase(SchedulerState.ENDED.toString())) {
+                                                        String message = "Job is in " + logAndStateTriple.getRight() + " state.";


It's almost the same message than above job state, need to fix it. And a question, do we need to expose those states to users? If yes, should they be present both, such as: -? #Resolved

wezhang · 2019-03-25T03:14:27Z

...ode-common/src/com/microsoft/azure/hdinsight/spark/common/CosmosServerlessSparkBatchJob.java

-                    if (err instanceof SparkJobFinishedException || err.getCause() instanceof SparkJobFinishedException) {
-                        return Observable.error(err);
+                .repeatWhen(ob -> ob.delay(GET_LIVY_URL_REPEAT_DELAY_MILLISECONDS, TimeUnit.MILLISECONDS))
+                .takeUntil(batchResp -> isJobEnded(batchResp) || StringUtils.isNotEmpty(getLivyAPI(batchResp)))


How about if getJobState() always returns null? #Resolved

Throw an exception in the new iteration.

In reply to: 268477836 [](ancestors = 268477836)

wezhang · 2019-03-25T07:02:34Z

...ode-common/src/com/microsoft/azure/hdinsight/spark/common/CosmosServerlessSparkBatchJob.java

@@ -204,6 +214,11 @@ public String getJobUuid() {
                })
                .flatMap(job ->
                        getSparkBatchJobRequest()
+                                .flatMap(batchResp ->


May try to use one getSparkBatchJobRequest() here

wezhang

LGTM! Shipit! Thanks Rui!

With this PR, - Serverless job can successfully run and finish when backend is stable. - Pop up Spark master UI web page when serverless job is starting - Warn user that kill serverless job action is invalid when the job is in `Finalizing` or `Ended` state

t-rufang requested review from konjac, wezhang and lcadzy March 21, 2019 17:21

wezhang reviewed Mar 22, 2019

View reviewed changes

t-rufang force-pushed the test_serverless branch from 25132ec to 009c92b Compare March 22, 2019 14:20

wezhang added this to the v3.20.0 milestone Mar 25, 2019

wezhang reviewed Mar 25, 2019

View reviewed changes

t-rufang force-pushed the test_serverless branch from 009c92b to 7345219 Compare March 25, 2019 06:35

t-rufang requested a review from wezhang March 25, 2019 06:42

wezhang reviewed Mar 25, 2019

View reviewed changes

t-rufang added 4 commits March 25, 2019 18:19

Fix serverless job run failed related issues

3837eec

Update based on review comments

9640e39

update based on review comments

7a6ae02

Update based on review comments

db83716

t-rufang force-pushed the test_serverless branch from 7345219 to db83716 Compare March 25, 2019 10:19

Merge branch 'develop' into test_serverless

c73f46f

wezhang approved these changes Mar 26, 2019

View reviewed changes

t-rufang merged commit 9560f3e into microsoft:develop Mar 26, 2019

t-rufang deleted the test_serverless branch April 10, 2020 05:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix serverless job run failed related issues #2899

Fix serverless job run failed related issues #2899

t-rufang commented Mar 21, 2019

wezhang Mar 22, 2019 •

edited by t-rufang

Loading

wezhang Mar 22, 2019 •

edited by t-rufang

Loading

wezhang Mar 22, 2019 •

edited by t-rufang

Loading

wezhang Mar 22, 2019 •

edited by t-rufang

Loading

wezhang Mar 22, 2019 •

edited by t-rufang

Loading

wezhang Mar 22, 2019 •

edited by t-rufang

Loading

t-rufang Mar 22, 2019

wezhang Mar 22, 2019 •

edited by t-rufang

Loading

wezhang Mar 22, 2019 •

edited by t-rufang

Loading

wezhang Mar 22, 2019 •

edited by t-rufang

Loading

wezhang Mar 22, 2019 •

edited by t-rufang

Loading

wezhang Mar 25, 2019 •

edited by t-rufang

Loading

t-rufang Mar 25, 2019

wezhang Mar 25, 2019

wezhang left a comment

Fix serverless job run failed related issues #2899

Fix serverless job run failed related issues #2899

Conversation

t-rufang commented Mar 21, 2019

wezhang Mar 22, 2019 • edited by t-rufang Loading

Choose a reason for hiding this comment

wezhang Mar 22, 2019 • edited by t-rufang Loading

Choose a reason for hiding this comment

wezhang Mar 22, 2019 • edited by t-rufang Loading

Choose a reason for hiding this comment

wezhang Mar 22, 2019 • edited by t-rufang Loading

Choose a reason for hiding this comment

wezhang Mar 22, 2019 • edited by t-rufang Loading

Choose a reason for hiding this comment

wezhang Mar 22, 2019 • edited by t-rufang Loading

Choose a reason for hiding this comment

t-rufang Mar 22, 2019

Choose a reason for hiding this comment

wezhang Mar 22, 2019 • edited by t-rufang Loading

Choose a reason for hiding this comment

wezhang Mar 22, 2019 • edited by t-rufang Loading

Choose a reason for hiding this comment

wezhang Mar 22, 2019 • edited by t-rufang Loading

Choose a reason for hiding this comment

wezhang Mar 22, 2019 • edited by t-rufang Loading

Choose a reason for hiding this comment

wezhang Mar 25, 2019 • edited by t-rufang Loading

Choose a reason for hiding this comment

t-rufang Mar 25, 2019

Choose a reason for hiding this comment

wezhang Mar 25, 2019

Choose a reason for hiding this comment

wezhang left a comment

Choose a reason for hiding this comment

wezhang Mar 22, 2019 •

edited by t-rufang

Loading

wezhang Mar 22, 2019 •

edited by t-rufang

Loading

wezhang Mar 22, 2019 •

edited by t-rufang

Loading

wezhang Mar 22, 2019 •

edited by t-rufang

Loading

wezhang Mar 22, 2019 •

edited by t-rufang

Loading

wezhang Mar 22, 2019 •

edited by t-rufang

Loading

wezhang Mar 22, 2019 •

edited by t-rufang

Loading

wezhang Mar 22, 2019 •

edited by t-rufang

Loading

wezhang Mar 22, 2019 •

edited by t-rufang

Loading

wezhang Mar 22, 2019 •

edited by t-rufang

Loading

wezhang Mar 25, 2019 •

edited by t-rufang

Loading