procstat input shouldn't cache PID

[aabouzaid]

procstat reports for processes aren't in the conf file ... and they look totally random from the system.

Here is an example for some procs name:

udevadm
telegraf
sudo
sleep
pgrep
kworker/2:2
cleanup

It collect data for these procs for sometime (also a random time) and then it stops!

System info:

System: Ubuntu 14.04.4 LTS
Telegraf: 0.13.1
pgrep: 3.3.9 (default one that comes with Ubuntu)

Steps to reproduce:

It happens randomly.

Expected behavior:

It should collect data for procs in config file only.

Actual behavior:

It collects data for arbitrary procs (and no errors related to procstat in telegraf logs).

Additional info:

Conf file:
/etc/telegraf/telegraf.d/procstat.conf

###############################################################################
#                                  OUTPUTS                                    #
###############################################################################
[[outputs.influxdb]]
  # The full HTTP or UDP endpoint URL for your InfluxDB instance.
  # Multiple urls can be specified but it is assumed that they are part of the same
  # cluster, this means that only ONE of the urls will be written to each interval.
  # urls = ["udp://localhost:8089"] # UDP endpoint example
  urls = ["http://10.10.10.10:8086"] # required
  # The target database for metrics (telegraf will create it if not exists)
  database = "proc_status" # required
  # Precision of writes, valid values are n, u, ms, s, m, and h
  # note: using second precision greatly helps InfluxDB compression
  precision = "s"

  # Connection timeout (for the connection with InfluxDB), formatted as a string.
  # If not provided, will default to 0 (no timeout)
  timeout = "30s"
  username = "proc_status"
  password = "*****************"
  # Set the user agent for HTTP POSTs (can be useful for log differentiation)
  # user_agent = "telegraf"
  # Set UDP payload size, defaults to InfluxDB UDP Client default (512 bytes)
  # udp_payload = 512

  # This metrics only will go to processes database.
  namepass = ["procstat*"]


###############################################################################
#                                  INPUTS                                     #
###############################################################################
[[inputs.procstat]]
  exe = "salt-minion"

[[inputs.procstat]]
  exe = "sshd"

[sparrc]

that is very odd, do you have any steps to reproduce? Are you sure there are no other conflicting configuration files? do you see any patterns, such as PID overlap or anything?

[aabouzaid]

I tried to find any pattern, but I couldn't find any (e.g. if there is another Telegraf plugin is enabled with procstat or so, but it happens in different servers and envs).

But while I'm checking if there is anything with PIDs, I found the exe name is different of process_name for the same PID (in different times).

time,                  exe,     host,       pid,      process_name
-----------------------------------------------------------------------
2016-08-11T04:03:20Z,  "sshd",  "xserver",  "13778",  "sshd"
2016-08-11T04:03:30Z,  "sshd",  "xserver",  "13778",  "java"
--
2016-08-16T01:04:50Z,  "sshd",  "xserver",  "13778",  "java"
2016-08-16T12:10:00Z,  "sshd",  "xserver",  "13778",  "telegraf"

Reuse of the PID when a process finishes is normal. So I think there is a time space between procstat gets/assigns name of process and its PID.

[sparrc]

I see, in that case I think I know what the bug is. The procstat plugin caches the PID because it assumes that it's not going to be changing. It does this to avoid having to query the system every interval for the pid with the given filter.

I think the only solution will be to query every interval for the PID.

[aabouzaid]

Great! Thanks @sparrc :-)

[sparrc]

closed by #2206

[sparrc]

reopening because #2206 is being reverted because it breaks the cpu_usage metric

see #2479