Skip to content
This repository has been archived by the owner on Nov 1, 2023. It is now read-only.

agent & supervisor heartbeat needs an initial jitter #386

Closed
bmc-msft opened this issue Dec 11, 2020 · 1 comment · Fixed by #387
Closed

agent & supervisor heartbeat needs an initial jitter #386

bmc-msft opened this issue Dec 11, 2020 · 1 comment · Fixed by #387
Labels
enhancement New feature or request

Comments

@bmc-msft
Copy link
Contributor

bmc-msft commented Dec 11, 2020 •
edited by mgreisen
Loading

The heartbeat async thread needs to start at a random interval within in the periodicity of the heartbeat.

As is, if we start ~3000 nodes at once, we have a recurring storm roughly once every 5 minutes when all of the nodes check in.

AB#43103051
@bmc-msft bmc-msft added the enhancement New feature or request label Dec 11, 2020
@ghost ghost added the Needs: triage label Dec 11, 2020
@bmc-msft
Copy link
Contributor Author

Note, the API endpoint handles it fine, but we'd be better served if it wasn't required.

@bmc-msft bmc-msft mentioned this issue Dec 11, 2020
Merged
@ghost ghost closed this as completed in #387 Jan 4, 2021
ghost pushed a commit that referenced this issue Jan 4, 2021
@bmc-msft
add delayed start to heartbeats (#387)
3441790 
Adds a random initial jitter the size of the heartbeat periodicity to prevent heartbeats storming the service when we launch 3000 nodes roughly at the same time.

Fixes #386
@ghost ghost locked as resolved and limited conversation to collaborators Feb 3, 2021
This issue was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

add delayed start to heartbeats bmc-msft/onefuzz
2 participants
@stishkin @bmc-msft