Consider tracking whether the machine would like a reboot (/var/run/reboot-required)

[andrewhowdencom]

Host operating system: output of uname -a

Debian / Ubuntu

node_exporter version: output of node_exporter -version

Are you running node_exporter in Docker?

Negative

What is the feature that you wish to see?

Debian flavour operating systems have the capability to perform unattended upgrades. These are fairly reliable; enough so that fault tolerant systems can be upgraded automatically. However, in some cases the upgrades are such that they require a machine restart. In some cases, it may not be desirable to restart the system on an ad-hoc basis -- instead, to schedule the restart at a later date.

They express this need by writing a file to /var/run/reboot-required. This is picked up in /etc/update-motd.d/, were it's existence is checked.

Alternatives

This might not be within the scope of what the node exporter should be querying. Currently, I plan to write a cron job that polls this file, and writes whether it exists to the textfile dir. Alternatively, something like osquery and the exporter therein might be suitable.

[SuperQ]

Having a gauge metric for the existence of /var/run/reboot-required is a good idea. It's simple enough and doesn't require root. My only hesitation is this is very distribution specific. Do any other distributions use this pattern?

We could possibly support it, but have the feature disabled by default due to the distribution specific nature of it.

[brian-brazil]

This has come up previously in various forms such as needing to do an apt-get upgrade. It's very distribution specific and we've previously said to use the textfile collector.

[SuperQ]

This specific one is much more appropriate as it's checking for file existence, rather than needing to execute or access a package database.

[SuperQ]

@andrewhowdencom Related, if you wanted to contribute a cleaned up version of #496, I would appreciate it.

[andrewhowdencom]

@SuperQ I had intended to do that with OSQuery and the relevant exporter: https://github.com/zwopir/osquery_exporter (not something I've gotten around to just yet). Still, doing it with bash seems like a trivial solution daily or something -- I'll take a look as I have time.

I am going to close this -- it seems reasonable to mark distro specific stuff out of bounds of the exporter. All else fails, i can always write "debian_exporter"

[SuperQ]

A debian_exporter helper script would be great to add to this repo in the textfile helper section. I recommend this because we typically don't want to have a scrape fork processes. Doing the updates via cron + textfile make things much safer.

[calestyo]

Just for the records... needrestart would be a distro-agnostic way to detect whether services, kernel, microcode or even, single (non-daemon) processes need restarts.

It even already provides a icinga/nagios check, so it should probably be easy to also craft something for prometheus.

[anarcat]

two things here:

1. the node exporter collectors have some support for this, exposed as a node_reboot_required flag in "yum" and "apt" configurations, so the request from the original post here is covered by that...
2. ... but as @calestyo hinted, that's actually not enough: it only tracks whether or not the system "thinks" it needs a reboot but typically misses a lot. for example, library upgrades which running binaries depend on will not mark a system as "needs reboot", which is where needrestart comes in: it scans the running processes and through some heuristics actually restarts relevant services or marks the other as requiring restart when it's unsafe to do automatically

I opened an issue in needrestart for this: liske/needrestart#291

I would welcome feedback on the proposed metrics names there... In my mind, this could be a cron job that punts things into the node exporter textfile directory, simply, but could be done in a number of ways...