process monitor throws warnings on postgresql: Warning: Process 123 disappeared while scanning

[ssbarnea]

process monitor throws warnings on postgresql: Warning: Process 123 disappeared while scanning

process
-------
- instance #0 [OK]
- instance DataDog/dd-agent#1 [WARNING]
Warning: Process 16039 disappeared while scanning
Warning: Process 16177 disappeared while scanning
Warning: Process 16178 disappeared while scanning
Warning: Process 16193 disappeared while scanning
Warning: Process 16194 disappeared while scanning
Warning: Process 16198 disappeared while scanning

      Warning: Process 15830 disappeared while scanning
      Warning: Process 15844 disappeared while scanning
  - instance DataDog/dd-agent#2 [OK]
  - instance DataDog/dd-agent#3 [OK]
  - instance DataDog/dd-agent#4 [OK]
  - instance DataDog/dd-agent#5 [OK]
  - instance DataDog/dd-agent#6 [OK]
  - instance DataDog/dd-agent#7 [OK]
  - instance DataDog/dd-agent#8 [OK]
  - instance DataDog/dd-agent#9 [OK]
  - Collected 43 metrics, 0 events & 11 service checks

Is is perfectly normal for postgresql to start and stop processes, and I see no reason why datadog would have to complain about that.

[yannmh]

Thanks for reporting @ssbarnea.

Our process check is very generic, and is not able to determine whether a process restart is legit or not. Therefore, this message is displayed with the WARNING level (not an ERROR).
Adding an option to disable this specific warning, would be an easy fix. Any thoughts @remh @degemer ?

[remh]

I think it's fine to keep it as a warning.
Closing this issue

[ssbarnea]

This bug will cause a DDOS on the logs if you have a big postgresql instance like the one I have, it does fill the logs with warnings.

[tomazstrukelj]

An option to disable this warning would certainly be a good idea, because if this process dynamic is a normal behaviour, it should not be listed with a warning.
In DataDog Infrastructure List Dashboard for example I see lots of warnings for php process that is rotating children all the time and it's hindering our ability to react to real warnings.

[remh]

Thanks for the feedback @tomazstrukeljdlabs . We could make an option to disable those warnings indeed. Reopening

[blackstar257]

Any update or workaround for this issue?

[thereforsunrise]

I've love an option to disable this. It is very spammy IMO.

[nethack47]

I know it's over a year old but this is quite relevant so any chance this will happen?

[chadharvey]

I would concur that a per-process warning suppression should be implemented. My experience is that as soon as you turn on a process check for any specific process, you lose the generic process count (TBH, not sure that worked before as we now have process checks one every host). To get around that, I had to add a process check like:

   {
        "exact_match": "False",
        "name": "all",
        "search_string": [
            ""
        ],
        "tags": [],
   }

(please forgive the json, artifact of how we do things)
This allows us to track the total number of process on the host.

This, much like the postgresql issue, generates a LOT of warnings. These show up as Integration issues on the Infrastructure list, and can hide other actual problems.

[michael-mcclintock-lu]

Would also be very appreciative of this as a configurable option. We're monitoring PHP processes and they are also prone to stopping while Datadog is scanning and it is quite spammy.

[arunsandu1]

we are monitoring cassandra processes and have the problem with the warnings. If the warnings are not actionable, is there any configuration to disable the warnings in the collector.log?

    process (5.21.0)
    ----------------
      - instance #0 [WARNING]
          Warning: No matching process 'cassandra-process' was found
      - instance DataDog/dd-agent#1 [WARNING]
          Warning: No matching process 'datastax-agent-process' was found
      - instance DataDog/dd-agent#2 [WARNING]
          Warning: No matching process 'opscenter-process' was found
      - Collected 3 metrics, 0 events & 3 service checks

Collector.log

2018-01-30 19:32:58 UTC | WARNING | dd.collector | checks.process(__init__.py:700) | No matching process 'cassandra-process' was found
2018-01-30 19:32:58 UTC | WARNING | dd.collector | checks.process(__init__.py:700) | No matching process 'datastax-agent-process' was found
2018-01-30 19:32:58 UTC | WARNING | dd.collector | checks.process(__init__.py:700) | No matching process 'opscenter-process' was found

[crabmusket]

Just ran into this when using process monitoring to track php-fpm, which manages its own child processes.

  - name: php-fpm
    search_string: ['php-fpm']
    exact_match: False

 Datadog's process integration is reporting:
Instance #process:php-fpm:488d3410896f9213[WARNING]:Process 17825 disappeared while scanning

A setting to disable this warning would be awesome.

[testworksau]

Also having the same issue, with php-cgi which scales processes up and down depending on the load of the server.

Is there a way I can contribute to your suggestion of disabling these warnings?

[albertvaka]

This warning got removed in #7045 which was released in Agent versions 6.22 and 7.22, please update your Agents to get the fix 😃 Closing this as done.