Use ps command that prints full command lines without errors

[nugend]

Description

Based on errors with the existing command, this should behave correctly

Motivation and Context

Fixes #1808

Have you tested this? If so, how?

Tested on colleagues machine. Tested command on our machine. Would need external validation about behavior with ps on more operating systems/versions.

[mention-bot]

@nugend, thanks for your PR! By analyzing the annotation information on this pull request, we identified @daveFNbuck, @erikbern and @steenzout to be potential reviewers

[daveFNbuck]

LGTM

[Tarrasch]

Can you add a couple of lines of comments above this line as you find necessary? Example coming to my mind:

• What OS versions this have been tested for
• Links to various online man-pages for various OSes
• Summarize what behaviour these set of parameters give and how it fits with luigi.

I mean, this is fully optional, I only want you to include what you think will make it easier for the next person dealing with this code line.

[Tarrasch]

LGTM. But consider adding some docs. :)

[nugend]

Sorry, I don't really have much time for a few weeks and I understand that this is a very hazardous sort of patch. Maybe you could solicit some other contributors to test out the command on various OS's?

[daveFNbuck]

I tested it on my Ubuntu servers and OSX laptop. It matches the man pages for those too.

[Tarrasch]

Okay. I merge this to fix the bug. But I would appreciate if we could follow up where you document your research efforts as comments in the code. :)

[keitap]

Unfortunately this fix doesn't work well on busybox (e.g. Alpine Linux) because it uses the 'w' option that busybox's ps doesn't support. (similar to #1530 )
How about to use /proc/(pid)/cmdline if platform.system() == 'Linux'?

[Tarrasch]

@keitap, any solution that works sounds fine for me ^^

[nugend]

@keitap Doesn't that only work if you know the pid beforehand?

Ugh. I almost hate to mention this, but maybe luigi really needs to depend on psutil if this is an important functionality? That's supposed to be cross platform.

[Tarrasch]

I think this is an important functionality. But it would be nice if we could avoid a dependency. :)

[nugend]

Ok, well, should ps even be used at all? Would it better to walk /proc and catch FileNotFound exceptions?

[Tarrasch]

Oh, yea maybe procfs is the right thing here. I for one don't know. :)

[Tarrasch]

@daveFNbuck @nugend @keitap @erikbern, I have some alarming experience regarding the locking. I don't remember how slot it was before this patch. But I've upgraded to the latest version of luigi. And I experience that startups (just the lock checking!) takes 3 minutes. This needs a fix soon!

We should think if we can use procfs or some other algorithm for this. Alternatively find out why this is slow. I could experiment with this a bit further. But I would like to hear from you if you've seen any regressions.

[Tarrasch]

Hmm... this seem to be because my .pid file have 7027 entries in it for some reason... I'll investigate this further ....

[Tarrasch]

Oh this seems to be for many many files.

❯ wc -l /tmp/luigi/*                                                                                                                                           
   1892 /tmp/luigi/03446b0238b6510876b4425b09ed4915.pid
   2085 /tmp/luigi/0394d228c401cc829c75afaac48b37a1.pid
   1896 /tmp/luigi/04633b957affa2565e67230450ab59d5.pid
   2084 /tmp/luigi/047a05715ee5ee9fcfd748c08f1cf556.pid
   1901 /tmp/luigi/049fc7eb10a773ff47c4da2a287d88e4.pid
   1895 /tmp/luigi/04d258946980574465457efc20c9f5f3.pid
   1895 /tmp/luigi/0503079a80d05e1f83e3ffe11ea0b751.pid
   1897 /tmp/luigi/078d6e1853656612934393d0833e9dca.pid
   1901 /tmp/luigi/082cf3864ae947f18e4ddd52487af814.pid
   1900 /tmp/luigi/08ae7c0344667b28a706aa7ee94767a0.pid
   1897 /tmp/luigi/095195163887c4c6b44db29f9a4ce694.pid
   1900 /tmp/luigi/0da99d2a9b7a86ec64352c4548f50677.pid
   1900 /tmp/luigi/0e1466533f00a00177dca96aebca2522.pid
   1894 /tmp/luigi/0e98c56f7cfd5b8ff2a0bc7233603c09.pid
   2093 /tmp/luigi/0ecc28a934f3efd9141a5c0a81e831fd.pid
   1899 /tmp/luigi/0f5c8d615681e10d642490cabcb9ff7a.pid
    ....

This is clearly unrelated to this patch I see. Sorry for alarming. Though I'll work on getting something up that cleans the .pid files and send a PR by tomorrow.