[Bug]: Windows node has gone stale and no longer showing in the parent Netdata instance.

[mcdent]

Bug description

A windows node, running the prometheus endpoint is now showing as 'stale' in the netdata cloud (NC) this was added as a vnode. It used to show but for some reason now it does now.
It also does not show at the end of the list in the Linux node which is scraping its data. I can connect

Expected behavior

I should actually be seeing 2 windows nodes at the bottom of the screenshot on the right side. As you can see in the screenshot neither are showing.

Steps to reproduce

Try to view the windows node in the list under the linux host that is configured locally to run netdata.
This is the same output when viewing via the cloud login or going direct to the local IP of netdata.
I can also curl to the prometheus endpoint running on the windows host.

Screenshots

Error Logs

No response

Desktop

OS: MAcOS
Browser Firefox
Browser Version 123.0 64 bit

Additional context

No response

[hugovalente-pm]

@mcdent do you have the same issue if you open your Linux machine local agent dashboard?
Have you also tried to run the go.d.plugin in debug mode from the Linux machine? https://learn.netdata.cloud/docs/collecting-metrics/windows-systems/windows#debug-mode

[mcdent]

Thanks for the tip @hugovalente-pm. I recreated the windows.conf and vnode.conf files, with fresh UIDs and now there seems to be no errors in the debug mode.
My remaining problem is the node which is marked as 'stale' is not collecting/sending any data, despite me being able to curl to the prometheus metric endpoint from the linux host. It shows as stale still in the linux host dashboard.

Is the fact it is marked as stay preventing this?

One other thing, no hardware spec of the windows machine is showing, I assume this is a limitation of prometheus?

Thanks
Mike

[hugovalente-pm]

My remaining problem is the node which is marked as 'stale' is not collecting/sending any data, despite me being able to curl to the prometheus metric endpoint from the linux host. It shows as stale still in the linux host dashboard.

can you try just having just that node configured on your Linux machine and then run the debug mode? also try to access your Linux machine local agent dashboard?

Is the fact it is marked as stay preventing this?

Stale means your Linux node has past data for that node but it doesn't "find" any current data.
a gut feeling, could there be a clock sync issue on that machine?

One other thing, no hardware spec of the windows machine is showing, I assume this is a limitation of prometheus?

yes and no, atm we only get those specs for nodes where the agent is locally running there. we know we need to find a way to get this for remote Windows machines but it something currently on our backlog

[mcdent]

My remaining problem is the node which is marked as 'stale' is not collecting/sending any data, despite me being able to curl to the prometheus metric endpoint from the linux host. It shows as stale still in the linux host dashboard.

can you try just having just that node configured on your Linux machine and then run the debug mode? also try to access your Linux machine local agent dashboard?

Ok, doing this has now brought the errant asdf123 windows machine back. I have not tried adding the config file again for the windows box wiresx1, I will do that next?
Notice although the asdf123 is now reporting, it still shows as stale?

Is the fact it is marked as stay preventing this?

Stale means your Linux node has past data for that node but it doesn't "find" any current data. a gut feeling, could there be a clock sync issue on that machine?

I checked the time on both windows boxes and they are both set to automatic and the time seems to align with that from the linux box collecting the stats.

One other thing, no hardware spec of the windows machine is showing, I assume this is a limitation of prometheus?

yes and no, atm we only get those specs for nodes where the agent is locally running there. we know we need to find a way to get this for remote Windows machines but it something currently on our backlog

IS this why I am missing memory stats for one thing?

I also note that the average CPU utilisation does not seem to align with that of the Windows 11 pc?
Task manager on windows shows a fairly steady 20% cpu (whilst running some programs) however looking back over the same period on Netdata shows it as much lower?


Am I expecting too much from the windows clients? I don't really want to run the full agent on these boxes if possible.

Thanks
Mike

[mcdent]

Hmm, now I set the configs back I have ended up with 2 stale asdf123 windows nodes?

[mcdent]

Ok, lets update this here, think I have 3 days left on my trial and have been unsuccessful to add more than 1 windows node at a time.
I tried adding a completely new windows 10 pro machine, installed the prometheus endpoint via the instructions, I can curl to the endpoint just fine. Added the new windows node to the windows.conf and vnode.conf and restarted net data.
Now I see the last working windows node as stale and my new one shows up? What am I doing wrong?

[hugovalente-pm]

Can you share the vnode.conf file? What it seems is that the UUID is clashing since you seem to get the last one connected to Cloud and the other becomes stale - we saw this same issue when VMs are copied and for the new one a new UUID isn't generated. Btw, if you need a couple of more days to complete your tests we can extend the trial for a bit longer

…

On Sun, 17 Mar 2024, 17:08 Mike, ***@***.***> wrote: Ok, lets update this here, think I have 3 days left on my trial and have been unsuccessful to add more than 1 windows node at a time. I tried adding a completely new windows 10 pro machine, installed the prometheus endpoint via the instructions, I can curl to the endpoint just fine. Added the new windows node to the windows.conf and vnode.conf and restarted net data. Now I see the last working windows node as stale and my new one shows up? What am I doing wrong? Screenshot.2024-03-17.at.16.57.59.jpg (view on web) <https://github.com/netdata/netdata-cloud/assets/8769856/bff9a060-a39d-4b3c-9f7f-e8a34c72ed4b> Screenshot.2024-03-17.at.16.58.56.jpg (view on web) <https://github.com/netdata/netdata-cloud/assets/8769856/268a186b-e815-4858-a5ca-913bc5ba4104> — Reply to this email directly, view it on GitHub <#986 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATTNB4EKHS6JUCRG3XWXARDYYXEZBAVCNFSM6AAAAABEND3TT6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBSGU2DCNZSGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[hugovalente-pm]

seems there was already feedback on the possible issue, shared on Discord here

Remove the extra jobs, keep just one on top
That's an array in yaml

@mcdent if this works please update the result here

[mcdent]

Ok, the issue here was that I had a separate job entry for each host. They should all have been under the one job section. e.g.

My bad all sorted.