Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increased timeout value for delete history event task #5888

Merged
merged 7 commits into from
Apr 9, 2024

Conversation

timl3136
Copy link
Contributor

@timl3136 timl3136 commented Apr 8, 2024

What changed?
We increased the timeout value for delete history event task to dynamic config with default at 30s.

Why?
The timeout value for delete history event is too low causing large history deletion to constantly timeout

How did you test it?

Potential risks

Release notes

Documentation Changes

Copy link

codecov bot commented Apr 8, 2024

Codecov Report

Merging #5888 (c90219f) into master (65c4017) will decrease coverage by 0.08%.
The diff coverage is 100.00%.

❗ Current head c90219f differs from pull request most recent head 24b8346. Consider uploading reports for the commit 24b8346 to get more accurate results

Additional details and impacted files
Files Coverage Δ
service/history/task/timer_active_task_executor.go 72.00% <100.00%> (+0.49%) ⬆️
...ervice/history/task/timer_standby_task_executor.go 78.94% <100.00%> (+0.98%) ⬆️
...ervice/history/task/transfer_task_executor_base.go 81.73% <ø> (ø)

... and 10 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 65c4017...24b8346. Read the comment docs.

@coveralls
Copy link

coveralls commented Apr 8, 2024

Pull Request Test Coverage Report for Build 018ec3d7-42d3-4387-8cf7-9808097685ae

Details

  • 9 of 9 (100.0%) changed or added relevant lines in 3 files are covered.
  • 37 unchanged lines in 11 files lost coverage.
  • Overall coverage increased (+0.02%) to 67.363%

Files with Coverage Reduction New Missed Lines %
service/matching/taskReader.go 2 84.88%
common/persistence/sql/sqlplugin/postgres/task.go 2 73.4%
service/history/shard/context.go 2 66.7%
common/persistence/sql/sqlplugin/postgres/db.go 2 80.0%
service/history/task/transfer_active_task_executor.go 2 72.7%
common/membership/hashring.go 2 84.69%
service/matching/taskListManager.go 3 79.7%
service/history/task/transfer_standby_task_executor.go 4 86.6%
service/history/task/task_util.go 4 75.14%
common/task/fifo_task_scheduler.go 7 82.47%
Totals Coverage Status
Change from base Build 018ec2d0-2d55-4385-9005-feae08321986: 0.02%
Covered Lines: 98199
Relevant Lines: 145776

💛 - Coveralls

@@ -110,7 +110,10 @@ func (t *timerStandbyTaskExecutor) Execute(
case persistence.TaskTypeWorkflowBackoffTimer:
return t.executeWorkflowBackoffTimerTask(ctx, timerTask)
case persistence.TaskTypeDeleteHistoryEvent:
return t.executeDeleteHistoryEventTask(ctx, timerTask)
// special timeout for delete history event
deleteHistoryEventContext, deleteHistoryEventCancel := context.WithTimeout(context.Background(), taskDeleteHistoryEventContextTimeout)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know the existing ctx has 3 sec timeout and doesn't listen for queue shut down signals.
With the new 60s timeout for history deletions it is going to be more likely to end up hanging a shutdown. We should consider making these operations responsive to shut down signals.
@Shaddoll what do you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed offline. We will hook up Stop signals to task executor in a follow up PR.
In this PR, let's make this 60s dynamically configurable

common/dynamicconfig/constants.go Outdated Show resolved Hide resolved
@timl3136 timl3136 enabled auto-merge (squash) April 9, 2024 17:14
@timl3136 timl3136 merged commit cd14f34 into cadence-workflow:master Apr 9, 2024
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants