-
-
Notifications
You must be signed in to change notification settings - Fork 2.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WIP] capture: do not close tmpfile buffer, but redirect #6034
Conversation
df4e3a6
to
b9d7d91
Compare
3e38537
to
5042501
Compare
5042501
to
05405bb
Compare
this seems icky to me, imho we shouldnt make pytest internals that much worse to work around other peoples broken setups |
It is not really much worse, and has e.g. the benefit to re-use tmpfiles, which seems like a good idea in general to me. |
upon closer reading its indeed not that bad, but it gets trickier and trickier to follow i believe by taking in py.io.capture we might be able to simplify it overall + perhaps even enable stdio output as views into the files to safe ram on larger test-suites as follow up |
I think that's not wanted in general. And it appears to not have any enhancements? (IIRC it looks like pytest adopted it) |
5f1864d
to
7deb6d2
Compare
I agree at first glance seems complicated, but I have some suggestions which I believe can improve the code and leave it more readable (I suspect this is just a quick first version to see the reception and if it would work). |
@@ -77,19 +81,29 @@ def __init__(self, method): | |||
self._method = method | |||
self._global_capturing = None | |||
self._current_item = None | |||
self._atexit_funcs = [] # type: List[Callable] | |||
atexit.register(self._atexit_run) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the tmpfile management can be isolated in a separate class, which would greatly simplify the code:
class Streams(IntEnum):
STDIN = 0
STDOUT = 1
STDERR = 2
@attr.s
class TemporaryFilePool:
_tmpfiles = attr.ib(default=attr.factory(dict))
def obtain_tmpfile(self, stream_id):
try:
tmpfile = self._tmpfiles[stream_id]
except KeyError:
if stream_id == Streams.STDIN:
tmpfile = open(os.devnull, "r")
else:
f = TemporaryFile()
with f:
tmpfile = safe_text_dupfile(f, mode="wb+")
self._tmpfiles[stream_id] = tmpfile
assert not tmpfile.closed
return tmpfile
def close(self):
for f in self._tmpfiles.values():
f.close()
self._tmpfiles.clear()
We should then make CaptureManager
create and maintain the tmpfile_pool
, and pass it around to FDCaptureBinary
. CaptureManager
should then register the atexit
handle itself, so these 3 lines would become:
self._tmpfile_pool = TemporaryFilePool()
atexit.register(self._tmpfile_pool)
This design also allows us to change the pool implementation, so we could even possibly control it via a config variable and get back the old behavior (I'm not actually suggesting it, just mentioning that this is then possible).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh I also realize now that we don't respect basetemp
when creating TemporaryFiles
...
def _getcapture(self, method): | ||
if method == "fd": | ||
return MultiCapture(out=True, err=True, Capture=FDCapture) | ||
return MultiCapture(out=True, err=True, Capture=FDCapture, capman=self) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here we would pass self._tmpfile_pool
to the MultiCapture
objects.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think passing capman
is more universal.
(IIRC I am using this when auto-suspending on readin from stdin)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
more universal means more coupling here... I prefer to pass only the necessary. I also suggest making it mandatory, so we don't have to check for its existense first (might require a few test changes I suppose).
@@ -551,15 +573,33 @@ def __init__(self, targetfd, tmpfile=None): | |||
self.done = self._done | |||
if targetfd == 0: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this entire if/else
can now be just:
tmpfile = self._tmpfile_pool.obtain_tmpfile(self.targetfd)
I also wonder if we should force that all handles are closed with |
@nicoddemus |
What do you mean? |
I like the idea of displaying them order, others are not so keen. But to clarify, do you mean create a new stream (stdouterr) in addition to stdout/stderr capture, or replacing them? |
Haven't heard other opinions.
It would be a seperate one. |
I haven't heard much against it either, except from a colleague at work (which is a great developer).
Hmm not sure what you mean here, can you please clarify? Anyway I really like the "captured output" idea, but this seems like the topic for a separate PR/issue.
Yes, but I was just wondering, not sure if this is a good idea or not. 😁 |
7deb6d2
to
06f2154
Compare
Is this abandoned? |
@blueyed Any reason why you closed this merge request? |
Fixes #5502
TODO:
[ ] after mypy: no_implicit_optional = False #6036