Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider replacing ProcessCapture with something else #4668

Open
sschuberth opened this issue Nov 5, 2021 · 3 comments
Open

Consider replacing ProcessCapture with something else #4668

sschuberth opened this issue Nov 5, 2021 · 3 comments
Labels
enhancement Issues that are considered to be enhancements

Comments

@sschuberth
Copy link
Member

Regarding the getVersion() invocation mentioned by @mnonnenmacher and @sschuberth, I did some testing. I ran a simple test that executes this method in a loop. What I found out is that there is indeed a small memory leak caused by ProcessCapture calling deleteOnExit() on the temporary files it creates to capture the process output. deleteOnExit() causes the name of the file affected to be recorded in a static Set managed by the internal DeleteOnExitHook system class, so that it can be deleted later.

So each usage of the ProcessCapture class adds three file names to this set (for stdout, stderr, and the temporary directory). This could be cured by ensuring manually that the files are cleaned up, e.g. using a try-finally construct. However, my gut feeling is that the amount of memory wasted here is not really significant.

Originally posted by @oheger-bosch in #3091 (comment)

@sschuberth sschuberth added the enhancement Issues that are considered to be enhancements label Nov 5, 2021
@fviernau
Copy link
Member

fviernau commented Nov 9, 2021

Maybe slightly related: Some of the pip commands the analyzer runs may take quite long. The process capture swallows all output during execution so that no progress can be observed at all. The output is only provided after the execution has finished. This makes it IMO harder to debug. Maybe this can be solved together with migrating to something else.

sschuberth added a commit that referenced this issue May 9, 2022
The fact that `ProcessCapture` uses temporary files to work around the
64k buffering limit is an implementation detail that should not be
exposed to ease switching to another process builder alternative
eventually. Also see [1].

[1]: #4668

Signed-off-by: Sebastian Schuberth <sebastian.schuberth@bosch.io>
sschuberth added a commit that referenced this issue May 10, 2022
The fact that `ProcessCapture` uses temporary files to work around the
64k buffering limit is an implementation detail that should not be
exposed to ease switching to another process builder alternative
eventually. Also see [1].

[1]: #4668

Signed-off-by: Sebastian Schuberth <sebastian.schuberth@bosch.io>
porsche-rbieniek pushed a commit to porsche-rbieniek/ort that referenced this issue Jun 24, 2022
The fact that `ProcessCapture` uses temporary files to work around the
64k buffering limit is an implementation detail that should not be
exposed to ease switching to another process builder alternative
eventually. Also see [1].

[1]: oss-review-toolkit#4668

Signed-off-by: Sebastian Schuberth <sebastian.schuberth@bosch.io>
porsche-rbieniek pushed a commit to porsche-rbieniek/ort that referenced this issue Jun 27, 2022
The fact that `ProcessCapture` uses temporary files to work around the
64k buffering limit is an implementation detail that should not be
exposed to ease switching to another process builder alternative
eventually. Also see [1].

[1]: oss-review-toolkit#4668

Signed-off-by: Sebastian Schuberth <sebastian.schuberth@bosch.io>
porsche-rbieniek pushed a commit to porsche-rbieniek/ort that referenced this issue Jun 27, 2022
The fact that `ProcessCapture` uses temporary files to work around the
64k buffering limit is an implementation detail that should not be
exposed to ease switching to another process builder alternative
eventually. Also see [1].

[1]: oss-review-toolkit#4668

Signed-off-by: Sebastian Schuberth <sebastian.schuberth@bosch.io>
@sschuberth
Copy link
Member Author

Maybe this can be solved together with migrating to something else.

I totally agree that streaming of (parts of) stdout to logs would be very useful!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Issues that are considered to be enhancements
Projects
None yet
Development

No branches or pull requests

2 participants