Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the scheduler option HeartbeatInterval #956

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

pior
Copy link
Contributor

@pior pior commented Nov 4, 2024

Context

From #510:

Currently the tick interval of Scheduler.runHeartbeater() and the ttl of WriteSchedulerEntries() are both 5*time.Second
scheduler entries is prone to expire when redis is slow to respond

Redo of #896

Fixes #510

Changes

  • Add option SchedulerOpts.HeartbeatInterval
  • Change default scheduler heartbeat interval from 5s to 10s

scheduler.go Outdated
@@ -317,7 +335,7 @@ func (s *Scheduler) beat() {
entries = append(entries, e)
}
s.logger.Debugf("Writing entries %v", entries)
if err := s.rdb.WriteSchedulerEntries(s.id, entries, 5*time.Second); err != nil {
if err := s.rdb.WriteSchedulerEntries(s.id, entries, s.heartbeatInterval); err != nil {
Copy link
Collaborator

@kamikazechaser kamikazechaser Nov 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From the original issue:

scheduler entries are prone to expire when redis is slow to respond

Even though this just affects the inspector, there is a small chance that the SchedulerEntries may be empty because they expired before the next beat ran. I would propose it to be such that redisKey ttl > beat tick to reduce the chance of the entries being empty at any given time.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Absolutely, I did not want to tackle this in the same PR, but I think we should have ttl = s.heartbeatInterval * 2.

I don't have strong opinions on this, what option do you prefer:

  • ttl = s.heartbeatInterval * 2
  • ttl = s.heartbeatInterval + time.Second
  • Or a second option

Copy link
Collaborator

@kamikazechaser kamikazechaser Nov 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ttl = s.heartbeatInterval * 2

This sounds good.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

scheduler entries ttl maybe too short
2 participants