[SPARK-36030][SQL] Support DS v2 metrics at writing path #33239

viirya · 2021-07-07T02:52:02Z

What changes were proposed in this pull request?

We add the interface for DS v2 metrics in SPARK-34366. It is only added for reading path, though. This patch extends the metrics interface to writing path.

Why are the changes needed?

Complete DS v2 metrics interface support in writing path.

Does this PR introduce any user-facing change?

No. For developer, yes, as this adds metrics support at DS v2 writing path.

How was this patch tested?

Added test.

viirya · 2021-07-07T02:52:40Z

cc @cloud-fan

SparkQA · 2021-07-07T03:48:51Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45238/

SparkQA · 2021-07-07T04:23:38Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45238/

SparkQA · 2021-07-07T05:51:57Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45250/

SparkQA · 2021-07-07T06:23:54Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45250/

viirya · 2021-07-07T06:42:38Z

cc @maropu too

SparkQA · 2021-07-07T07:50:21Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45254/

SparkQA · 2021-07-07T08:23:42Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45254/

SparkQA · 2021-07-07T09:22:20Z

Test build #140740 has finished for PR 33239 at commit 05800b4.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ContinuousWriteRDD(var prev: RDD[InternalRow], writerFactory: StreamingDataWriterFactory,
case class WriteToContinuousDataSource(write: StreamingWrite, query: LogicalPlan,
case class WriteToContinuousDataSourceExec(write: StreamingWrite, query: SparkPlan,

SparkQA · 2021-07-07T11:22:00Z

Test build #140743 has finished for PR 33239 at commit f656335.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2021-07-08T01:46:23Z

I also ran a query using the CustomMetricsDataSource added in the test. The UI looks like:

viirya · 2021-07-09T05:50:23Z

@cloud-fan Do we consider this in 3.2 to make the API complete in this release?

viirya · 2021-07-15T22:12:26Z

gentle ping @cloud-fan

sql/core/src/test/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListenerSuite.scala

dongjoon-hyun

This PR looks reasonable to me, @viirya . I left a few comments about test cases.

It would be great if we can get your advice, @cloud-fan , @gengliangwang , @HeartSaVioR , @Ngone51 , @maropu , @HyukjinKwon , @sunchao , @huaxingao .

viirya · 2021-07-19T06:57:54Z

Thank you @dongjoon-hyun! I will address the comments.

gengliangwang · 2021-07-19T13:15:27Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatDataWriter.scala

    taskAttemptContext: TaskAttemptContext,
-    committer: FileCommitProtocol) extends DataWriter[InternalRow] {
+    committer: FileCommitProtocol,
+    customMetrics: Map[String, SQLMetric]) extends DataWriter[InternalRow] {


Shall we add tests for the changes in this File?

Let me try to add one.

I added custom metric for writing to InMemory table for test purpose. The tests are in FileFormatDataWriterMetricSuite.

gengliangwang · 2021-07-19T13:15:52Z

@viirya Thanks for the work! Overall this LGTM

SparkQA · 2021-07-20T08:59:18Z

Test build #141294 has finished for PR 33239 at commit fe9cc4e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class InMemorySimpleCustomMetric extends CustomMetric

SparkQA · 2021-07-20T09:52:42Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45827/

SparkQA · 2021-07-20T10:30:24Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45827/

SparkQA · 2021-07-20T10:58:07Z

Test build #141298 has finished for PR 33239 at commit bccc98b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gengliangwang

LGTM

SparkQA · 2021-07-20T12:12:02Z

Test build #141313 has finished for PR 33239 at commit f62f057.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2021-07-20T16:27:02Z

retest this please

viirya · 2021-07-20T16:41:48Z

Thanks @dongjoon-hyun @gengliangwang @sunchao. I will merge this after tests pass.

cc @cloud-fan

SparkQA · 2021-07-20T17:47:32Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45864/

SparkQA · 2021-07-20T18:22:51Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45864/

SparkQA · 2021-07-20T19:59:29Z

Test build #141350 has finished for PR 33239 at commit 25cc546.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2021-07-20T20:51:21Z

retest this please

SparkQA · 2021-07-20T21:07:27Z

Test build #141359 has finished for PR 33239 at commit 25cc546.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2021-07-20T21:51:36Z

retest this please

SparkQA · 2021-07-20T21:54:03Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45873/

SparkQA · 2021-07-20T22:08:38Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45876/

SparkQA · 2021-07-20T22:33:52Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45873/

SparkQA · 2021-07-20T22:43:24Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45876/

SparkQA · 2021-07-20T23:08:46Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45878/

SparkQA · 2021-07-20T23:49:24Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45878/

SparkQA · 2021-07-21T02:53:00Z

Test build #141362 has finished for PR 33239 at commit 25cc546.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2021-07-21T03:19:56Z

Thanks. Merging to master/3.2.

### What changes were proposed in this pull request? We add the interface for DS v2 metrics in SPARK-34366. It is only added for reading path, though. This patch extends the metrics interface to writing path. ### Why are the changes needed? Complete DS v2 metrics interface support in writing path. ### Does this PR introduce _any_ user-facing change? No. For developer, yes, as this adds metrics support at DS v2 writing path. ### How was this patch tested? Added test. Closes #33239 from viirya/v2-write-metrics. Authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by: Liang-Chi Hsieh <viirya@gmail.com> (cherry picked from commit 2653201) Signed-off-by: Liang-Chi Hsieh <viirya@gmail.com>

cloud-fan · 2021-07-21T04:19:52Z

late LGTM

github-actions bot added SQL STRUCTURED STREAMING labels Jul 7, 2021

viirya marked this pull request as draft July 7, 2021 02:52

This comment has been minimized.

Sign in to view

Add writer metrics.

05800b4

viirya force-pushed the v2-write-metrics branch from 245db51 to 05800b4 Compare July 7, 2021 04:47

Add test.

f656335

github-actions bot added the WEB UI label Jul 7, 2021

viirya changed the title ~~[WIP][SPARK-36030][SQL] Support DS v2 metrics at writing path~~ [SPARK-36030][SQL] Support DS v2 metrics at writing path Jul 7, 2021

viirya marked this pull request as ready for review July 7, 2021 06:41

dongjoon-hyun reviewed Jul 19, 2021

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListenerSuite.scala Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Jul 19, 2021

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListenerSuite.scala Outdated Show resolved Hide resolved

dongjoon-hyun approved these changes Jul 19, 2021

View reviewed changes

gengliangwang reviewed Jul 19, 2021

View reviewed changes

Use eventually.

454acb4

gengliangwang approved these changes Jul 20, 2021

View reviewed changes

dongjoon-hyun approved these changes Jul 20, 2021

View reviewed changes

Fix typo on variable name.

25cc546

viirya closed this in 2653201 Jul 21, 2021

FabianMeiswinkel mentioned this pull request Oct 12, 2023

[FEATURE REQ]Add custom metrics (for example for RU consumption) in Cosmso DB Spark connector Azure/azure-sdk-for-java#37172

Closed

viirya deleted the v2-write-metrics branch December 27, 2023 18:27

[SPARK-36030][SQL] Support DS v2 metrics at writing path #33239

[SPARK-36030][SQL] Support DS v2 metrics at writing path #33239

Uh oh!

Conversation

viirya commented Jul 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

viirya commented Jul 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

SparkQA commented Jul 7, 2021

Uh oh!

SparkQA commented Jul 7, 2021

Uh oh!

SparkQA commented Jul 7, 2021

Uh oh!

SparkQA commented Jul 7, 2021

Uh oh!

viirya commented Jul 7, 2021

Uh oh!

SparkQA commented Jul 7, 2021

Uh oh!

SparkQA commented Jul 7, 2021

Uh oh!

SparkQA commented Jul 7, 2021

Uh oh!

SparkQA commented Jul 7, 2021

Uh oh!

viirya commented Jul 8, 2021

Uh oh!

viirya commented Jul 9, 2021

Uh oh!

viirya commented Jul 15, 2021

Uh oh!

Uh oh!

Uh oh!

dongjoon-hyun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

viirya commented Jul 19, 2021

Uh oh!

gengliangwang Jul 19, 2021

Choose a reason for hiding this comment

Uh oh!

viirya Jul 19, 2021

Choose a reason for hiding this comment

Uh oh!

viirya Jul 20, 2021

Choose a reason for hiding this comment

Uh oh!

gengliangwang commented Jul 19, 2021

Uh oh!

SparkQA commented Jul 20, 2021

Uh oh!

SparkQA commented Jul 20, 2021

Uh oh!

SparkQA commented Jul 20, 2021

Uh oh!

SparkQA commented Jul 20, 2021

Uh oh!

gengliangwang left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 20, 2021

Uh oh!

viirya commented Jul 20, 2021

Uh oh!

viirya commented Jul 20, 2021

Uh oh!

SparkQA commented Jul 20, 2021

Uh oh!

SparkQA commented Jul 20, 2021

Uh oh!

SparkQA commented Jul 20, 2021

viirya commented Jul 7, 2021 •

edited

Loading

viirya commented Jul 7, 2021 •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading