FreeBSD 10 zombie processes

[rvagg]

I'm regularly having to forcefully restart at least one of our FreeBSD machines (most recently I recall doing the DigitalOcean one, I believe that I've had to do the Rackspace one more than a few times as well, the Joyent ones seem happy though).

I noticed that test-digitalocean-freebsd10-x64-1 was one of the machines that's been causing grief these last couple of days, taking way too long to build and test, so I logged in and checked it out and found this, and my guess is that this is a precursor to the machine fully locking up.

iojs     5568   0.0  0.0       0      0  -  Z    20Dec17      0:00.01 <defunct>
iojs     5569   0.0  0.0       0      0  -  Z    20Dec17      0:00.01 <defunct>
iojs    16849   0.0  0.0       0      0  -  Z    27Dec17      0:00.00 <defunct>
iojs    16850   0.0  0.0       0      0  -  Z    27Dec17      0:00.00 <defunct>
iojs    24070   0.0  0.0   14676      0  -  IW   -            0:00.00 gmake run-ci -j 2
iojs    24122   0.0  0.0   14676      0  -  IW   -            0:00.00 gmake
iojs    24165   0.0  0.1   22868   1668  -  I    27Nov17      0:05.32 gmake -C out BUILDTYPE=Release V=1
iojs    29512   0.0  0.0   14676      0  -  IW   -            0:00.00 gmake run-ci -j 2
iojs    29564   0.0  0.0   14676      0  -  IW   -            0:00.00 gmake
iojs    29607   0.0  0.1   26964   1668  -  I     8Jan18      0:05.79 gmake -C out BUILDTYPE=Release V=1
iojs    39709   0.0  0.0   14676      0  -  IW   -            0:00.00 gmake run-ci -j 2
iojs    39761   0.0  0.0   14676      0  -  IW   -            0:00.00 gmake
iojs    39804   0.0  0.1   22868   1668  -  I    19Dec17      0:03.53 gmake -C out BUILDTYPE=Release V=1
iojs    40603   0.0  0.0   14676      0  -  IW   -            0:00.00 gmake run-ci -j 2
iojs    40655   0.0  0.0   14676      0  -  IW   -            0:00.00 gmake
iojs    40698   0.0  0.1   26964   1668  -  I    23Nov17      0:02.80 gmake -C out BUILDTYPE=Release V=1
iojs    43121   0.0  0.0       0      0  -  Z    27Nov17      0:00.00 <defunct>
iojs    43122   0.0  0.0       0      0  -  Z    27Nov17      0:00.00 <defunct>
iojs    45238   0.0  0.0   14676      0  -  IW   -            0:00.00 gmake run-ci -j 2
iojs    45290   0.0  0.0   14676      0  -  IW   -            0:00.00 gmake
iojs    45335   0.0  0.1   26964   1668  -  I    19Dec17      0:10.88 gmake -C out BUILDTYPE=Release V=1
iojs    49547   0.0  0.0       0      0  -  Z    19Dec17      0:00.03 <defunct>
iojs    49549   0.0  0.0       0      0  -  Z    19Dec17      0:00.00 <defunct>
iojs    50158   0.0  0.0       0      0  -  Z     8Jan18      0:00.00 <defunct>
iojs    50159   0.0  0.0       0      0  -  Z     8Jan18      0:00.00 <defunct>
iojs    58529   0.0  0.0   14676      0  -  IW   -            0:00.00 gmake run-ci -j 2
iojs    58581   0.0  0.0   14676      0  -  IW   -            0:00.00 gmake
iojs    58624   0.0  0.1   26964   1668  -  I    16Nov17      0:04.10 gmake -C out BUILDTYPE=Release V=1
iojs    66891   0.0  0.0       0      0  -  Z    23Nov17      0:00.01 <defunct>
iojs    66897   0.0  0.0       0      0  -  Z    23Nov17      0:00.00 <defunct>
iojs    74366   0.0  0.0   14676      0  -  IW   -            0:00.00 gmake run-ci -j 2
iojs    74420   0.0  0.0   14676      0  -  IW   -            0:00.00 gmake
iojs    74465   0.0  0.1   22868   1668  -  I    31Dec17      0:03.28 gmake -C out BUILDTYPE=Release V=1
iojs    75921   0.0  0.2   17876   3120  -  I     8:43AM      0:00.01 bash /tmp/jenkins7754032494571783635.sh
iojs    75924   0.0  0.1   14676   2984  -  I     8:43AM      0:00.07 gmake run-ci -j 2
iojs    75976   0.0  0.1   14676   3020  -  I     8:43AM      0:00.02 gmake
iojs    76019   0.0  0.7   26964  14848  -  I     8:43AM      0:04.27 gmake -C out BUILDTYPE=Release V=1
iojs    78438   0.0  0.0   14676      0  -  IW   -            0:00.00 gmake run-ci -j 2
iojs    78490   0.0  0.0   14676      0  -  IW   -            0:00.00 gmake
iojs    78533   0.0  0.1   22868   1668  -  I    27Dec17      0:02.25 gmake -C out BUILDTYPE=Release V=1
iojs    80616   0.0  0.0       0      0  -  Z    19Dec17      0:00.02 <defunct>
iojs    80617   0.0  0.0       0      0  -  Z    19Dec17      0:00.02 <defunct>
iojs    82174   0.0  0.0   14676      0  -  IW   -            0:00.00 gmake run-ci -j 2
iojs    82226   0.0  0.0   14676      0  -  IW   -            0:00.00 gmake
iojs    82269   0.0  0.1   22868   1668  -  I    19Dec17      0:02.31 gmake -C out BUILDTYPE=Release V=1
iojs    85475   0.0  0.0       0      0  -  Z    31Dec17      0:00.00 <defunct>
iojs    85476   0.0  0.0       0      0  -  Z    31Dec17      0:00.00 <defunct>
iojs    88500   0.0  0.0   14676      0  -  IW   -            0:00.00 gmake run-ci -j 2
iojs    88552   0.0  0.0   14676      0  -  IW   -            0:00.00 gmake
iojs    88595   0.0  0.1   22868   1668  -  I    20Dec17      0:05.28 gmake -C out BUILDTYPE=Release V=1
iojs    89287   0.0  0.0       0      0  -  Z    19Dec17      0:00.00 <defunct>
iojs    89288   0.0  0.0       0      0  -  Z    19Dec17      0:00.00 <defunct>
iojs    93134   0.0  0.0       0      0  -  Z    27Dec17      0:00.00 <defunct>
iojs    93135   0.0  0.0       0      0  -  Z    27Dec17      0:00.00 <defunct>
iojs    93637   0.0  0.0   14676      0  -  IW   -            0:00.00 gmake run-ci -j 2
iojs    93689   0.0  0.0   14676      0  -  IW   -            0:00.00 gmake
iojs    93732   0.0  0.1   26964   1668  -  I    27Dec17      0:03.23 gmake -C out BUILDTYPE=Release V=1
iojs    97397   0.0  0.0       0      0  -  Z    16Nov17      0:00.00 <defunct>
iojs    97399   0.0  0.0       0      0  -  Z    16Nov17      0:00.00 <defunct>
iojs    97847   0.0  0.0       0      0  -  Z     8:47AM      0:00.00 <defunct>
iojs    97848   0.0  0.0       0      0  -  Z     8:47AM      0:00.00 <defunct>

Load average was not particularly high either, these are zombies doing not very much.

No idea what to do about this, how to test it, suggestions would be welcome. In the meantime I've restarted the machine but I imagine it'll start building up this cruft again over time.

@nodejs/platform-freebsd

[maclover7]

Would hopefully be fixed by #591, moving discussion to there