Feature: Health checks windows service #13388

deblasis · 2022-06-07T17:43:39Z

Description

To provide windows service health check functionality addressing the issue #10637

Testing & Reproduction steps

The PR is Draft because I would like you guys to point me to your preferred way of mocking syscalls and I will update the feature branch with the relevant tests.

Links

Issue: windows service health check #10637
PRD (kinda 🙂): https://gist.github.com/deblasis/b6dd645a8bd5e2b3c9b95bb30bbb238c

PR Checklist

updated test coverage
external facing docs updated
not a security concern

Amier3 · 2022-06-08T20:26:52Z

Hey @deblasis

Super excited to see this PR come through, thank you! After looking ( and asking ) around it doesn't look like we have any established pattern for mocking syscalls. You're free to add the tests in whenever you get a chance 👍 😄

deblasis · 2022-06-09T14:51:49Z

Hey @deblasis

Super excited to see this PR come through, thank you! After looking ( and asking ) around it doesn't look like we have any established pattern for mocking syscalls. You're free to add the tests in whenever you get a chance 👍 😄

Hi @Amier3 !

I have added syscall mocking and testing, I guess this is now finally ready for review! 😁

GOOS=windows go test -tags=windows ./agent/checks -run TestCheck_OSService -v

deblasis · 2022-06-09T14:58:23Z

agent/checks/os_service_windows.go

+	defer func() {
+		errDisconnect := m.Disconnect()
+		if isHealthy || errDisconnect == nil || err != nil {
+			return
+		}
+		//unreachable at the moment but we might want to log this error. leaving here for code-review
+		err = errDisconnect
+	}()


I am doing this because I think it might be important to at least log errors if there are any at this point (after changing the if condition of course) and I wanted to raise the question with you.

If not, the code can be simplified with a simple

defer func () { m.Disconnect() }()

without capturing the error.
Same applies below for service.Close().

From a quick glance I think the fail conditions for Disconnect and Close are unlikely and can be ignored.

In the worst case scenario the process holding the ServiceManager open doesn't get cleaned up but if the OS is degraded to the point where it can't close there may be worse problems 😅

I'd vote for a simple defer but it's not a strong opinion by any means.

Same comment here as the one about getWindowsSvc. What would happen if OpenService fail and we execute Disconnect anyway?

trujillo-adam

I reviewed the documentation updates and made a few suggestions to improve clarity.

website/content/docs/discovery/checks.mdx

website/content/api-docs/agent/check.mdx

jkirschner-hashicorp · 2022-06-15T16:23:15Z

@deblasis : for context, our education/docs team has performed a review (above). The engineering review still needs to be conducted! I'd recommend waiting for an engineering review before making changes so you can address all the feedback in one go.

github-actions · 2022-08-15T01:38:30Z

This pull request has been automatically flagged for inactivity because it has not been acted upon in the last 60 days. It will be closed if no new activity occurs in the next 30 days. Please feel free to re-open to resurrect the change if you feel this has happened by mistake. Thank you for your contributions.

…cks_windows_service Signed-off-by: Alessandro De Blasis <alex@deblasis.net>

kisunji

Thank you for this large chunk of design and implementation! I'm not sure if we'll be able to implement Windows machines for CI to test this but I'll try it out locally on my desktop and ask the team how we want to handle this windows-specific feature.

I left a few minor comments that are non-blocking.

agent/checks/check.go

agent/checks/os_service_windows.go

kisunji · 2022-08-17T15:02:30Z

agent/checks/os_service_windows.go

+	defer func() {
+		errDisconnect := m.Disconnect()
+		if isHealthy || errDisconnect == nil || err != nil {
+			return
+		}
+		//unreachable at the moment but we might want to log this error. leaving here for code-review
+		err = errDisconnect
+	}()


From a quick glance I think the fail conditions for Disconnect and Close are unlikely and can be ignored.

In the worst case scenario the process holding the ServiceManager open doesn't get cleaned up but if the OS is degraded to the point where it can't close there may be worse problems 😅

I'd vote for a simple defer but it's not a strong opinion by any means.

agent/checks/os_service_windows.go

kisunji · 2022-08-17T15:12:15Z

agent/checks/check.go

+	ServiceID     structs.ServiceID
+	OSService     string
+	Interval      time.Duration
+	Timeout       time.Duration


I don't see Timeout being used anywhere. Is it worth implementing a timeout error? Unlike gRPC or script checks I would assume OS service polling shouldn't take too long and it's not worth enforcing.

dhiaayachi

Thank you for working on this @deblasis!
I left few questions and an improvement suggestion but nothing substantial to add to @kisunji review.

dhiaayachi · 2022-08-23T14:22:09Z

agent/checks/os_service_windows.go

+		err = nil
+		isHealthy = true
+	case svc.Paused, svc.Stopped:
+		err = ErrOSServiceStatusCritical


maybe we can wrap the service state inside ErrOSServiceStatusCritical to help user have what state is failing the check.

dhiaayachi · 2022-08-23T14:22:12Z

agent/checks/os_service_windows.go

+	service := win.getWindowsSvc(serviceName, svcHandle)
+	defer func() {
+		errClose := service.Close()
+		if isHealthy || errClose == nil || err != nil {
+			return
+		}
+		//unreachable at the moment but we might want to log this error. leaving here for code-review
+		err = errClose
+	}()
+	status, err := service.Query()
+	if err != nil {
+		return fmt.Errorf("error querying service status: %w", err)
+	}


This is a bit confusing for me. I don't know how in details the windows library work, but looking to this, getWindowsSvc will only allocate the struct and don't do any access, right? in that case what will happen if Query fail for some reason but we still call service.Close in the defer? will service.Close error?

dhiaayachi · 2022-08-23T14:26:30Z

agent/checks/os_service_windows.go

+	defer func() {
+		errDisconnect := m.Disconnect()
+		if isHealthy || errDisconnect == nil || err != nil {
+			return
+		}
+		//unreachable at the moment but we might want to log this error. leaving here for code-review
+		err = errDisconnect
+	}()


Same comment here as the one about getWindowsSvc. What would happen if OpenService fail and we execute Disconnect anyway?

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>

…cks_windows_service Signed-off-by: Alessandro De Blasis <alex@deblasis.net>

kisunji · 2022-08-30T17:42:37Z

@deblasis would you be able to rebase or merge from main? It should have fixes for the failing integration tests

…cks_windows_service

kisunji · 2022-09-06T13:35:34Z

Thanks! Tests are all passing. Will make some time this week to test on Windows

kisunji · 2022-09-16T19:33:08Z

Apologies for the delay; I didn't have time this week but will make sure to test next week. I will also post a comment on the original github issue to see if there is anyone who would like to test out the feature.

kisunji · 2022-09-20T21:40:29Z

I discovered a few issues while testing:

We need to initialize checkOSServices otherwise there is a panic due to assigning to a nil map
(not a blocker) UI does not recognize OS as a type of check so it appears as a "Serf Check" in the UI.
I am seeing the following in the logs:

2022-09-20T17:29:34.057-0400 [DEBUG] agent: Check failed: check=service-check error="error accessing service: The handle is invalid."

kisunji · 2022-09-20T21:43:45Z

Reproduction steps:

// config/nodecheck.hcl
check = {
  name = "node-check"
  os_service = "AdobeARMService"
  interval = "10s"
}

// config/svccheck.hcl
check = {
  name = "service-check"
  os_service = "AdobeARMService"
  service_id = "hello"
  interval = "10s"
}

# powershell as admin
.\consul.exe agent -dev -config-dir=".\config"

Confirming that AdodeARMService is running

PS C:\Users\Chris> Get-Service -Name "AdobeARMService"

Status   Name               DisplayName
------   ----               -----------
Running  AdobeARMservice    Adobe Acrobat Update Service

deblasis · 2022-09-21T12:19:14Z

Hi @kisunji! Sorry about that. But I have good news 👇

We need to initialize checkOSServices otherwise there is a panic due to assigning to a nil map

Addressed in fc0dd92

(not a blocker) UI does not recognize OS as a type of check so it appears as a "Serf Check" in the UI.

I have noticed that I missed some properties and the corresponding generated mappings (fixed in 461b42e) but however I believe there's a bug elsewhere, I have added another check to... check 🙂 (repro config below) and I am seeing serf in there as well:

Since I was looking at the UI, I noticed the lack of message so I have added a "Healthy" message with 5719fd6

I am seeing the following in the logs:

2022-09-20T17:29:34.057-0400 [DEBUG] agent: Check failed: check=service-check error="error accessing service: The handle is invalid."

Addressed in f440966

I have tested the above with:

// config/nodecheck.hcl
check = {
  name = "node-check"
  os_service = "Dhcp"
  interval = "10s"
}

// config/h2ping.hcl
check = {
  id = "h2ping-check"
  name = "h2ping"
  h2ping = "localhost:22222"
  interval = "10s"
  h2ping_use_tls = false
}

kisunji · 2022-09-21T14:44:56Z

Ah it must be a UI bug then. Thanks for the quick updates, will try to test again soon

jkirschner-hashicorp · 2022-09-21T17:11:52Z

@johncowen: any ideas on why newer health check types (os_service from this PR, h2ping introduced a few months ago) would show up as a "Serf" health check on the UI?

johncowen · 2022-09-21T17:18:00Z

If the Type field in the HTTP response that the UI uses is empty then we assume that is a serf check. Possibly or possibly not useful UI PR #10194

I would guess that the new healthchecks are coming out with empty Types in the HTTP responses.

Umm separate thing but if we have new healthcheck types then the UI code might need to be aware of those for menus etc. not totally sure if we inspect those or if they are hardcoded.

At the very least UI should add these possible values to our mock API I think

deblasis · 2022-09-21T21:16:33Z

Hi @johncowen ! Nice seeing you again around here (you reviewed a PR of mine back in the day) :)

If the Type field in the HTTP response that the UI uses is empty then we assume that is a serf check. Possibly or possibly not useful UI PR #10194

I would guess that the new healthchecks are coming out with empty Types in the HTTP responses.

I can confirm that this is the case, the Type is empty in the HTTP responses even for "older" checks like h2ping.

trujillo-adam

Had a few suggestions and questions w/r/t the docs. You did a really good job on the main description of the OS service check, BTW.

trujillo-adam · 2022-09-22T15:36:19Z

website/content/api-docs/agent/check.mdx

@@ -245,6 +245,8 @@ The check sends datagrams to the value specified at the interval specified in th
 If the datagram is sent successfully or a timeout is returned, the check is set to the `passing` state.
 The check is logged as `critical` if the datagram is sent unsuccessfully.

+- `OSService` `(string: "")` - Specifies the name of a service to check that is running as an OS-level service; either Windows Services on Windows or SystemD services on Unix.


Suggested change

- `OSService` `(string: "")` - Specifies the name of a service to check that is running as an OS-level service; either Windows Services on Windows or SystemD services on Unix.

- `OSService` `(string: "")` - Specifies an OS-level service to check. You can specify either `Windows Services` on Windows or `SystemD` services on Unix.

The current wording reads as if we are checking the actual Windows Service or SystemD application, but stated in a convoluted way. I think we might mean, though, that this option tells Consul that the service it is configured to check runs as a Windows Service or SystemD service. Is that correct?

Correct. Currently, this feature works only on windows but the main pieces for making it work with SystemD are in place. The design was intended to offer the same look-n-feel to the user independently of the implementation details/OS.

I just realized that I accepted this by mistake and I feel like it should be reverted and/or rephrased to indicate that it's the name of the OS-specific service that goes into that.

website/content/docs/ecs/configuration-reference.mdx

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>

deblasis · 2022-09-23T06:49:52Z

@trujillo-adam:

Had a few suggestions and questions w/r/t the docs. You did a really good job on the main description of the OS service check, BTW.

Suggestions accepted! Thanks but @jkirschner-hashicorp and yourself helped a lot already giving it the first pass a few weeks ago.
My initial version was not that clear. Now it looks pretty good. 👍

trujillo-adam

One more tweak

website/content/api-docs/agent/check.mdx

trujillo-adam · 2022-09-23T20:32:07Z

website/content/api-docs/agent/check.mdx

@@ -245,6 +245,8 @@ The check sends datagrams to the value specified at the interval specified in th
 If the datagram is sent successfully or a timeout is returned, the check is set to the `passing` state.
 The check is logged as `critical` if the datagram is sent unsuccessfully.

+- `OSService` `(string: "")` - Specifies an OS-level service to check. You can specify either `Windows Services` on Windows or `SystemD` services on Unix.


Suggested change

- `OSService` `(string: "")` - Specifies an OS-level service to check. You can specify either `Windows Services` on Windows or `SystemD` services on Unix.

- `OSService` `(string: "")` - Specifies the name of an OS-level service. Configure this option if the service you intend to check runs as an OS-level service. You can specify either `Windows Services` on Windows or `SystemD` services on Unix.

I think trying to combine the action to specify the name and the action of performing the check (if that makes sense) is what's causing confusion for me. We should state them separately. First, what the param is for. Second, what the param means in the context of the overall API call. What do you think?

deblasis · 2022-10-11T09:06:39Z

👍

…

On Mon, 29 Aug 2022 at 18:05, Chris S. Kim ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In agent/agent.go <#13388 (comment)>: > + if prev := a.checkOSServices[cid]; prev != nil { + prev.Stop() + } I wouldn't mind if you just removed the line from the docker case as well — Reply to this email directly, view it on GitHub <#13388 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHAEQNW57SXDCXXOIMI2FPLV3TUVJANCNFSM5YDYBKAQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

deblasis · 2022-10-14T11:35:16Z

I think it should be OK now. Shall we🚢 it ? 😄

kisunji · 2022-10-17T13:21:05Z

@deblasis I've tested this successfully on Windows so this is good to 🚢 !

deblasis added 2 commits June 7, 2022 18:25

docs: added os_service

4de260f

feat: windows service health checks

b59c19b

github-actions bot added pr/dependencies PR specifically updates dependencies of project theme/config Relating to Consul Agent configuration, including reloading type/docs Documentation needs to be created/updated/clarified labels Jun 7, 2022

chore: changelog

2cd5201

deblasis mentioned this pull request Jun 7, 2022

windows service health check #10637

Closed

deblasis added 2 commits June 9, 2022 15:48

tests: added syscall mocking and tests for Check_OSService

af083cc

tests: removed redundant probe test

28f19e4

deblasis marked this pull request as ready for review June 9, 2022 14:51

deblasis requested a review from a team as a code owner June 9, 2022 14:51

deblasis commented Jun 9, 2022

View reviewed changes

deblasis changed the title ~~[WIP] Feature: Health checks windows service~~ Feature: Health checks windows service Jun 9, 2022

lint: conversion

06304bf

trujillo-adam reviewed Jun 15, 2022

View reviewed changes

website/content/docs/discovery/checks.mdx Outdated Show resolved Hide resolved

website/content/api-docs/agent/check.mdx Outdated Show resolved Hide resolved

github-actions bot added the meta/stale Automatically flagged for inactivity by stalebot label Aug 15, 2022

jkirschner-hashicorp requested review from kisunji and dhiaayachi August 15, 2022 05:10

deblasis added 2 commits August 15, 2022 08:09

Merge remote-tracking branch 'hashicorp/main' into feature/health-che…

ab611ea

…cks_windows_service Signed-off-by: Alessandro De Blasis <alex@deblasis.net>

Merge remote-tracking branch 'hashicorp/main' into feature/health-che…

5dee555

…cks_windows_service Signed-off-by: Alessandro De Blasis <alex@deblasis.net>

kisunji approved these changes Aug 17, 2022

View reviewed changes

dhiaayachi reviewed Aug 23, 2022

View reviewed changes

deblasis and others added 4 commits August 28, 2022 17:26

Update website/content/docs/discovery/checks.mdx

61f6fa9

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>

Update website/content/api-docs/agent/check.mdx

f0f0421

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>

fix(OSServiceCheck): fixes following code-review

f634e36

Merge remote-tracking branch 'hashicorp/main' into feature/health-che…

f3437ea

…cks_windows_service Signed-off-by: Alessandro De Blasis <alex@deblasis.net>

deblasis added 2 commits August 29, 2022 17:53

fix(agent): removed redundant check on prev. running check

c0d647d

fix(agent): removed redundant code in docker check as well

26cc56b

Merge remote-tracking branch 'hashicorp/main' into feature/health-che…

60c7c83

…cks_windows_service

deblasis added 4 commits September 21, 2022 09:25

fix(agent): uninitialized map panic error

fc0dd92

fix(checks): os_service lifecycle bugfix

f440966

fix(checks): os_service OK message in output

5719fd6

fix(check): added missing OSService props

461b42e

deblasis requested a review from kisunji September 21, 2022 12:19

trujillo-adam reviewed Sep 22, 2022

View reviewed changes

deblasis and others added 2 commits September 23, 2022 07:43

Update website/content/api-docs/agent/check.mdx

a39d034

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>

Update website/content/docs/ecs/configuration-reference.mdx

5bba2a3

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>

trujillo-adam reviewed Sep 23, 2022

View reviewed changes

website/content/api-docs/agent/check.mdx Outdated Show resolved Hide resolved

trujillo-adam reviewed Sep 23, 2022

View reviewed changes

Update website/content/api-docs/agent/check.mdx

5f99f57

kisunji merged commit 3d2dfff into hashicorp:main Oct 17, 2022

jkirschner-hashicorp mentioned this pull request Oct 18, 2023

Docs bug - os_service is not implemented on linux. #19037

Open

	- `OSService` `(string: "")` - Specifies the name of a service to check that is running as an OS-level service; either Windows Services on Windows or SystemD services on Unix.
	- `OSService` `(string: "")` - Specifies an OS-level service to check. You can specify either `Windows Services` on Windows or `SystemD` services on Unix.

Feature: Health checks windows service #13388

Feature: Health checks windows service #13388

Conversation

deblasis commented Jun 7, 2022 • edited Loading

Description

Testing & Reproduction steps

Links

PR Checklist

Amier3 commented Jun 8, 2022

deblasis commented Jun 9, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trujillo-adam left a comment

Choose a reason for hiding this comment

jkirschner-hashicorp commented Jun 15, 2022

github-actions bot commented Aug 15, 2022

kisunji left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dhiaayachi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kisunji commented Aug 30, 2022

kisunji commented Sep 6, 2022

kisunji commented Sep 16, 2022

kisunji commented Sep 20, 2022

kisunji commented Sep 20, 2022

deblasis commented Sep 21, 2022

kisunji commented Sep 21, 2022

jkirschner-hashicorp commented Sep 21, 2022 • edited Loading

johncowen commented Sep 21, 2022

deblasis commented Sep 21, 2022

trujillo-adam left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

deblasis commented Sep 23, 2022 • edited Loading

trujillo-adam left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

deblasis commented Oct 11, 2022 via email

deblasis commented Oct 14, 2022

kisunji commented Oct 17, 2022 • edited Loading

deblasis commented Jun 7, 2022 •

edited

Loading

deblasis commented Jun 9, 2022 •

edited

Loading

jkirschner-hashicorp commented Sep 21, 2022 •

edited

Loading

deblasis commented Sep 23, 2022 •

edited

Loading

kisunji commented Oct 17, 2022 •

edited

Loading