Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-50378][SS] Add custom metric for tracking spent for proc initial state in transformWithState #48913

Closed
wants to merge 1 commit into from

Conversation

anishshri-db
Copy link
Contributor

What changes were proposed in this pull request?

Add custom metric for tracking spent for proc initial state in transformWithState

Why are the changes needed?

Adds tracking for time spent in populating initial state

Does this PR introduce any user-facing change?

No

How was this patch tested?

Added unit tests

[info] Run completed in 2 minutes, 38 seconds.
[info] Total number of tests run: 22
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 22, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.

Was this patch authored or co-authored using generative AI tooling?

No

@anishshri-db anishshri-db changed the title [SPARK-50378] Add custom metric for tracking spent for proc initial state in transformWithState [SPARK-50378][SS] Add custom metric for tracking spent for proc initial state in transformWithState Nov 21, 2024
@anishshri-db
Copy link
Contributor Author

cc - @HeartSaVioR - PTAL, thx !

Copy link
Contributor

@HeartSaVioR HeartSaVioR left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
Will we have a separate PR to address this in PySpark impl? I guess the execution model may not be exactly the same, but if feasible, better to be consistent.

@HeartSaVioR
Copy link
Contributor

HeartSaVioR commented Nov 22, 2024

I'll leave it to you; I won't block this PR just to ensure consistency.

Thanks! Merging to master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants