-
Notifications
You must be signed in to change notification settings - Fork 814
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Low/No Disk space caused the Agent to crash and not recover. #2223
Conversation
e79a195
to
449e94a
Compare
It seems like the default case when calling |
@JohnLZeller I agree that we normally do use it to run commands where we want an output, but from a library point of view - and this being a library - I don't really want to enforce that kind of behavior were empty output is automatically akin to an error. |
@JohnLZeller actually nevermind.... you're right, the function is |
449e94a
to
292992e
Compare
…xception is raised.
292992e
to
3d22e91
Compare
@@ -37,6 +37,10 @@ def get_subprocess_output(command, log, shell=False, stdin=None): | |||
|
|||
stdout_f.seek(0) | |||
output = stdout_f.read() | |||
|
|||
if output_expected and output == "": |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: could change output == ""
to not output
. That way it'll catch ""
and None
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think read() always returns something... but I agree None is more pythonic.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, you're right :) definitely a nit
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
don't know if maybe we should .strip()
before testing if the string is empty, some rude programs sometimes output newlines for instance.
Looking pretty good. I wonder if it'd be worth defining a small Exception class, and raising that. This way we can catch a specific exception rather than all of them. Maybe something like this: Then you can just: And later can catch |
|
||
if Platform.is_freebsd(platf): | ||
output, _, _ = get_subprocess_output(['sysctl', 'hw.ncpu'], log) | ||
systemStats['cpuCores'] = int(output.split(': ')[1]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not your code, but seems like this could be written as:
if Platform.is_linux(platf):
...
elif Platform.is_darwin(platf) or Platform.is_freebsd(platf):
...
c9516b6
to
d31d4dd
Compare
if Platform.is_darwin(platf) or Platform.is_freebsd(platf): | ||
output, _, _ = get_subprocess_output(['sysctl', 'hw.ncpu'], log) | ||
systemStats['cpuCores'] = int(output.split(': ')[1]) | ||
except Exception as e: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ditto, Could we do a catch SubprocessOutputEmptyError:
here instead?
LGTM, other than the notes 👍 |
[subprocess] wrapping calls in relevant try-catch blocks.
6b06e13
to
661e9ca
Compare
Low/No Disk space caused the Agent to crash and not recover.
It seems like a low/no disk space can cause the TemporaryFile calls made in
subprocess_output
to "fail" - and by fail we mean empty output that can lead to IndexErrors, etc, when parsing the output. In general we have to be ready to expect an empty string as output.