Skip to content

Spark 3.5: Display write metrics on SQL UI #11340

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
manuzhang wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Spark 3.5: Display write metrics on SQL UI #11340

manuzhang wants to merge 1 commit into from

Conversation

@manuzhang
Copy link
Member

@manuzhang manuzhang commented Oct 17, 2024

No description provided.

@manuzhang manuzhang force-pushed the spark-write-metrics branch 3 times, most recently from 8a561bd to 316dc40 Compare October 17, 2024 15:03
@manuzhang
Copy link
Member Author

manuzhang commented Oct 18, 2024

Add number of total data files to write command AppendData

CleanShot 2024-10-18 at 11 31 55@2x

@manuzhang manuzhang force-pushed the spark-write-metrics branch from 316dc40 to 99d4b65 Compare October 18, 2024 04:44
manuzhang
manuzhang commented Nov 8, 2024
View reviewed changes
spark/v3.5/spark/src/main/scala/org/apache/spark/sql/MetricsUtils.scala Outdated

object MetricsUtils {

def postDriverMetrics(sparkContext: SparkContext, metricValues: java.util.Map[CustomMetric, Long]): Unit = {
Copy link
Member Author

@manuzhang manuzhang Nov 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not needed if we can support it at Spark side.

Copy link
Contributor

@Fokko Fokko Jan 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see that one got in :)

Copy link
Member Author

@manuzhang manuzhang Jan 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but it will only be available in Spark 4+

@manuzhang manuzhang mentioned this pull request Nov 8, 2024
Closed
@github-actions
Copy link

github-actions bot commented Dec 9, 2024

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions bot added the stale label Dec 9, 2024
@github-actions github-actions bot removed the stale label Dec 13, 2024
@github-actions
Copy link

github-actions bot commented Jan 12, 2025

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions bot added the stale label Jan 12, 2025
@manuzhang manuzhang added not-stale and removed stale labels Jan 12, 2025
@wypoon
Copy link
Contributor

wypoon commented Jan 15, 2025

@manuzhang I am happy to see that someone is working on adding write-side Iceberg metrics to the Spark SQL UI!
I realize that this is still in a draft state, but I have some questions/suggestions.
Do you plan to add metrics only to append operations? It would be good to see them for other operations, such as delete and overwrite.
Would added data files be more useful than total data files? or we could show both?
For delete and overwrite operations, I think it would be useful to see removed data files, added delete files and removed delete files.

wypoon
wypoon reviewed Jan 15, 2025
View reviewed changes
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/metrics/TotalDataFiles.java Outdated

@Override
public String description() {
return "number of total data files";
Copy link
Contributor

@wypoon wypoon Jan 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't sound right.
"total" implies a number, so I think the description can just be "total data files".
If you really want to include "number", then "total number of data files".

@manuzhang
Copy link
Member Author

manuzhang commented Jan 16, 2025

@wypoon I plan to add metrics for all write operations, but I'd like to get the interfaces right at first. I'm not sure whether this is the best way to propagate a metricsReporter. Any thoughts?

    if (this.table instanceof BaseTable) {
      this.metricsReporter = new InMemoryMetricsReporter();
      ((BaseTable) this.table).combineMetricsReporter(metricsReporter);
    }

@manuzhang manuzhang force-pushed the spark-write-metrics branch from f91d3f5 to 2e8e0c0 Compare May 2, 2025 15:54
@manuzhang manuzhang marked this pull request as ready for review May 2, 2025 15:55
@manuzhang manuzhang force-pushed the spark-write-metrics branch from 2e8e0c0 to fa49efc Compare May 3, 2025 15:24
@manuzhang manuzhang requested review from Fokko, aokolnychyi and wypoon and removed request for wypoon May 4, 2025 15:32
@manuzhang manuzhang force-pushed the spark-write-metrics branch from fa49efc to 2643854 Compare May 12, 2025 10:29
@manuzhang
Copy link
Member Author

manuzhang commented May 14, 2025

Snapshot of Spark SQL UI after latest update

CleanShot 2025-05-14 at 17 54 48@2x

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Fokko Fokko Awaiting requested review from Fokko

@aokolnychyi aokolnychyi Awaiting requested review from aokolnychyi

1 more reviewer

@wypoon wypoon wypoon left review comments

Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

core not-stale spark

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@manuzhang @wypoon @Fokko