Skip to content

Conversation

@ericm-db
Copy link
Contributor

What changes were proposed in this pull request?

This PR introduces a new configuration parameter spark.sql.streaming.stateStore.maintenanceForceShutdownTimeout to control the timeout for force shutdown operations in the StateStore maintenance thread pool.

Previously, the force shutdown timeout was hardcoded to 60 seconds in the MaintenanceThreadPool.stop() method. This PR makes it configurable to allow better control over graceful vs forceful shutdown behavior in different deployment scenarios.

Why are the changes needed?

Different streaming workloads and operational environments may require different force shutdown timeout values:

  • Some environments may need longer timeouts to allow in-flight maintenance operations to complete properly
  • Other environments may prefer shorter timeouts to speed up shutdown sequences
  • Making this configurable provides operators with more control over the tradeoff between clean shutdown and shutdown speed

Does this PR introduce any user-facing change?

No

How was this patch tested?

  • Existing tests continue to pass
  • The default value maintains backward compatibility with the previous hardcoded behavior

Was this patch authored or co-authored using generative AI tooling?

No

@ericm-db ericm-db changed the title Add configurable force shutdown timeout for StateStore maintenance thread pool [SPARK-54675] Add configurable force shutdown timeout for StateStore maintenance thread pool Dec 10, 2025
@ericm-db ericm-db changed the title [SPARK-54675] Add configurable force shutdown timeout for StateStore maintenance thread pool [SPARK-54675][SS] Add configurable force shutdown timeout for StateStore maintenance thread pool Dec 10, 2025
Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM (Pending CIs).

@ericm-db ericm-db changed the title [SPARK-54675][SS] Add configurable force shutdown timeout for StateStore maintenance thread pool [SPARK-54675][SPARK-54655][SS] Add configurable force shutdown timeout for StateStore maintenance thread pool Dec 10, 2025
Copy link
Contributor

@anishshri-db anishshri-db left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm pending green CI

@HyukjinKwon
Copy link
Member

Merged to master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants