Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Clustering stability #879

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

Clustering stability #879

wants to merge 8 commits into from

Conversation

carlhoerberg
Copy link
Member

WHAT is this pull request doing?

Improving clustering

HOW can this pull request be tested?

@carlhoerberg
Catch and explicity reraise IO::Errors in etcd
9fbfd38 
Otherwise when an Etcd method yielded, and that inner call raised
IO::Error that was interpreted as a Etcd error.
@carlhoerberg
hard exit if unhandled exception while Following a leader
fecad09
@carlhoerberg
Extra logging related to Following
cc61f5c
@carlhoerberg
document another etcd bug
db8c971
@carlhoerberg
Start etcd lease keepalive after won election
45a608c 
Apprently there's no need to update lease TTL until the election is won
@carlhoerberg
Refactor Leadership lease keepalive
1fc63fe
@carlhoerberg
always have a read timeout on Follower sockets
3fb5f9c 
we want to timeout when waiting for acks, if the follower is
unresponsive
@carlhoerberg
fixup! Refactor Leadership lease keepalive
6894ace
@carlhoerberg carlhoerberg marked this pull request as ready for review December 17, 2024 00:55
@carlhoerberg carlhoerberg requested a review from a team as a code owner December 17, 2024 00:55
@viktorerlingsson
Copy link
Member

Ran some tests on this and sometimes (like 3 out of 10 times) I'm getting this error when shutting down the leader, and the follower will not take over as leader. Haven't looked further into why.

2024-12-18T09:56:32.813755Z  INFO lmq.data_dir_lock Data directory locked by 'PID 243709 @ viktor-lenovo'
2024-12-18T09:56:32.813812Z  INFO lmq.data_dir_lock Waiting for file lock to be released
2024-12-18T09:56:32.813941Z ERROR lmq.etcd Lost leadership
Lease 7587883508156972415 expired (LavinMQ::Etcd::Error)
  from src/lavinmq/etcd.cr:71:7 in 'lease_ttl'
  from src/lavinmq/etcd.cr:121:15 in 'keepalive_loop'
  from src/lavinmq/etcd.cr:101:9 in '->'
  from /usr/share/crystal/src/fiber.cr:143:11 in 'run'
  from /usr/share/crystal/src/fiber.cr:95:34 in '->'
  from ???

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@carlhoerberg @viktorerlingsson