Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refine yawollet health check in keepailved #280

Merged
merged 8 commits into from
Jan 22, 2024
Merged

Refine yawollet health check in keepailved #280

merged 8 commits into from
Jan 22, 2024

Conversation

maboehm
Copy link
Collaborator

@maboehm maboehm commented Jan 22, 2024

Currently, the keepalived priority is influenced by a track_file which is supposed to represent that the LBM matches the latest revision. This allows for a rolling update, since all older instances will have a lower priority.

However, the track file is not always removed correctly, for example if the yawollet cannot reach the API-Server at all. So this change replaces the file-based check with a health endpoint, which is checked by keepalived via track_script. This now covers 3 conditions:

  • is the yawollet running correctly (if now wget gets no response/timeouts)
  • are the informers/caches synced
  • is the LBM the current revision
  • Is the LBM condition heartbeat up-to-date.

The last check is essentially a check if we can reach the API-Server. We could call the /healthz endpoint, but that might cause unnecessary load. By checking the conditions that the yawollet should be writing itself, we check both that we can write and read (watch) successfully from the API-Server without causing additional calls.

Copy link
Member

@timebertt timebertt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few nits

internal/healthz/heartbeat.go Outdated Show resolved Hide resolved
internal/healthz/heartbeat.go Outdated Show resolved Hide resolved
cmd/yawollet/main.go Outdated Show resolved Hide resolved
cmd/yawollet/main.go Outdated Show resolved Hide resolved
@timebertt timebertt changed the title WIP: Refine Yawollet health check in keepailved Refine Yawollet health check in keepailved Jan 22, 2024
@timebertt timebertt marked this pull request as ready for review January 22, 2024 13:27
timebertt
timebertt previously approved these changes Jan 22, 2024
Copy link
Member

@timebertt timebertt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@maboehm maboehm changed the title Refine Yawollet health check in keepailved Refine yawollet health check in keepailved Jan 22, 2024
internal/helper/loadbalancermachine.go Outdated Show resolved Hide resolved
timebertt
timebertt previously approved these changes Jan 22, 2024
@timebertt timebertt merged commit 4651708 into main Jan 22, 2024
2 checks passed
@timebertt timebertt deleted the yawollet-health branch January 22, 2024 15:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants