Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add support for health-check flag #1271

Merged
merged 14 commits into from
Jul 28, 2022
Merged

feat: add support for health-check flag #1271

merged 14 commits into from
Jul 28, 2022

Conversation

enocom
Copy link
Member

@enocom enocom commented Jul 19, 2022

No description provided.

@enocom enocom requested a review from a team July 19, 2022 21:21
@enocom
Copy link
Member Author

enocom commented Jul 19, 2022

Reading some more about healthchecks here (and in the linked items), I'm a bit worried about including dial attempts in a readiness check, but given that was the initial implementation, I'm inclined to perserve it for now. The liveness probe, I think, should also return to the original implementation where it's mostly a process health check.

Copy link
Collaborator

@hessjcg hessjcg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have some questions about the concurrency behavior in the health check. Checks should respond immediately, not wait for an application state before responding.

internal/healthcheck/healthcheck.go Show resolved Hide resolved
internal/proxy/proxy.go Show resolved Hide resolved
internal/proxy/proxy.go Show resolved Hide resolved
@enocom enocom requested a review from kurtisvg July 26, 2022 18:02
@enocom
Copy link
Member Author

enocom commented Jul 26, 2022

Copy link
Collaborator

@hessjcg hessjcg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work on the test coverage. LGTM.

cmd/root_test.go Show resolved Hide resolved
cmd/root.go Show resolved Hide resolved
cmd/root.go Outdated Show resolved Hide resolved
cmd/root.go Outdated Show resolved Hide resolved
cmd/root.go Outdated

// Start the HTTP server is either Prometheus or the health check was
// enabled.
if cmd.prometheusNamespace != "" || cmd.healthCheck {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe just add "useHttpServe" var somewhere? there maybe be more things in the future, it'll be easier to just set a variable to true when each flag is called rather than update the if statement here

Copy link
Member Author

@enocom enocom Jul 27, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we have to take different actions for prometheus vs health check, we do need two separate values. Nonetheless, looking again at prometheus, I see the namespace is optional, so I'll update the flag to be a boolean (and add a namespace flag separately), which cleans this up slightly.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On second thought, I'll make this change as a separate PR to keep things focused here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So what I was trying to suggest was that the need for an http server is probably orthogonal to which specific httpPorts have been specified.

So something like this:

if cmd.prometheusNamespace != "" {
    cmd.needHttpServer = true
    // other health check stuff
}
if cmd.healthCheck {
    cmd.needHttpServer = true
    // other health check stuff
}

if cmd.needHttpServer {
     // start http server
}

(Consider this a nit)

cmd/root.go Outdated
select {
case <-ctx.Done():
// Give the HTTP server a second to shutdown cleanly.
ctx2, _ := context.WithTimeout(context.Background(), time.Second)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any reason not to call defer cancel here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope. Will do.

internal/healthcheck/healthcheck.go Show resolved Hide resolved
}

if open, max := c.proxy.ConnCount(); max > 0 && open == max {
c.logger.Errorf("[Health Check] Readiness failed: %v", errMaxConnections)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: consider reporting the max number of connections already made

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll update the error to capture that information.

internal/proxy/proxy.go Outdated Show resolved Hide resolved
internal/proxy/proxy.go Show resolved Hide resolved
internal/proxy/proxy.go Show resolved Hide resolved
@enocom
Copy link
Member Author

enocom commented Jul 27, 2022

I had to rebase v2 onto main with this PR open. As a result, there are a lot of commits here that already exist on v2. But the diff is correct and will cleanly apply onto v2 now.

Update: I rebased onto v2 and removed the merge commit to get a clean result.

enocom and others added 12 commits July 27, 2022 15:29
Co-authored-by: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com>
Co-authored-by: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com>
Co-authored-by: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com>
cmd/root.go Outdated

// Start the HTTP server is either Prometheus or the health check was
// enabled.
if cmd.prometheusNamespace != "" || cmd.healthCheck {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So what I was trying to suggest was that the need for an http server is probably orthogonal to which specific httpPorts have been specified.

So something like this:

if cmd.prometheusNamespace != "" {
    cmd.needHttpServer = true
    // other health check stuff
}
if cmd.healthCheck {
    cmd.needHttpServer = true
    // other health check stuff
}

if cmd.needHttpServer {
     // start http server
}

(Consider this a nit)

enocom added a commit to GoogleCloudPlatform/alloydb-auth-proxy that referenced this pull request Jul 28, 2022
@enocom enocom merged commit bdc1139 into v2 Jul 28, 2022
@enocom enocom deleted the v2-healthcheck branch July 28, 2022 17:11
enocom added a commit to GoogleCloudPlatform/alloydb-auth-proxy that referenced this pull request Jul 28, 2022
enocom added a commit that referenced this pull request Aug 23, 2022
enocom added a commit that referenced this pull request Aug 29, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants