Too many opened file handles to spotbugs plugins

[svilenvul]

We are running the spotbugs-maven-plugin programmatically in a multi-module Maven project. This is why it is difficult to describe the exact spotbugs maven plugin configuration, this is why I will attach part of the Maven debug logs, where this can be seen:

spotbugs-config.txt

While trying to set up a Jenkins job, we faced ‘Too many open files’ problem causing the build to fail.

We started investigating the problem and found out that there is a huge number of opened files (spotbugs plugins).

I attach here the output of the following command ls -l /proc/MAVEN_PID/fd/ > openfiles.txt:

openfiles.txt

They are all located in the target directory.

Used spotbugs-maven-plugin/3.1.3 with spotbugs 3.1.2

[KengoTODA]

note from my investigation:

• ICodeBase,IClassPath is AutoCloseable.
• IClassPath is responsible to close all ICodeBase instances inside it
• FindBugs2#clearCaches() closes generated IClassPath
• It seems that this plugin doesn't invoke FindBugs2#clearCaches() for now

So I guess that we should invoke this FindBugs2#clearCaches() method, or create classloader to GC FindBugs2.classpath that is singleton field?

I will test following patch later.

diff --git a/spotbugs/src/main/java/edu/umd/cs/findbugs/FindBugs2.java b/spotbugs/src/main/java/edu/umd/cs/findbugs/FindBugs2.java
index dc061d2a8..1be3e9691 100644
--- a/spotbugs/src/main/java/edu/umd/cs/findbugs/FindBugs2.java
+++ b/spotbugs/src/main/java/edu/umd/cs/findbugs/FindBugs2.java
@@ -1185,7 +1185,7 @@ public class FindBugs2 implements IFindBugsEngine {
 
 
         FindBugs.runMain(findBugs, commandLine);
-
+        findBugs.clearCaches();
     }

UPDATE

• findBugs.clearCaches() are already invoked in FindBugs2#execute() which is called by FindBugs#runMain(...).

[KengoTODA]

spotbugs/spotbugs#589 is similar with this issue, but this issue's target is mainly jar files in classpath to scan.

[KengoTODA]

@svilenvul let me confirm: are your Maven uses -T option, to run modules in parallel?

[KengoTODA]

WIP: https://github.com/spotbugs/spotbugs/compare/fix-resource-leaks

[svilenvul]

@svilenvul let me confirm: are your Maven uses -T option, to run modules in parallel?

No, currently it is not being used.

[KengoTODA]

We started investigating the problem and found out that there is a huge number of opened files (spotbugs plugins).

Almost all opened file isn't not spotbugs plugin, but libraries in aux classpath.

In current architecture, spotbugs keeps all jar files opened during analysis (because IClassPath is keep opened), so we should face problem like this. To fix this issue, I think we need to change internal architecture to handle libraries in aux & app classpaths as stream.

[famod]

Has there been any progress? We are now seeing this frequently on our Jenkins CI server (we are using -T and there can also be concurrent builds).

[hazendaz]

This issue is likely related to #535 that has existed in this code base since its original inception with findbugs. Has to do with the -T and it getting locks as system.in was being called. This is to be removed in upcoming release 4.7.3.1 within the week.

[hazendaz]

closing as believe fixed with 4.7.3.1. If still an issue, please open a new ticket based off current data.