Parallel build without forking in a multi-module leads to spotbugs getting stuck #488
Comments
|
@blacelle This is a strange one, we run our integration tests on github actions in parallel and those all work. I presume you cannot share what you are running more than you have here. If that is the case, can you see if it is feasible to produce a small sample that has this issue or possibly look in our readme at how we run integration tests and give any pointers into what we are doing that may differ that we might try to cause same issue? |
Indeed, this happens with a private project. I'm not able to reproduce with public projects with similar configuration. I see in https://github.com/apache/ant/blob/master/src/main/org/apache/tools/ant/Project.java#L1300 that given piece of code may rely on System.in, which may explains something getting stuck. Given the 3 ExecuteJava seems to be relying on the same underlying BufferedInputStream, I would guess there are conflicting with each others, as I would not expect these 3 modules to interact in such a way. Maybe one module consumed part of the input, and the other module in waiting indefinitely for some input consumed by the other thread. I suppose this is not reproduced in spotbugs own integration tests due to some race-condition |
|
https://github.com/spotbugs/spotbugs/blob/master/spotbugs/src/main/java/edu/umd/cs/findbugs/TextUICommandLine.java#L409 seems to confirms the 3 modules relies on System.in. Maybe we lack a synchronized block around https://github.com/spotbugs/spotbugs/blob/master/spotbugs/src/main/java/edu/umd/cs/findbugs/TextUICommandLine.java#L297, to easily circumvent such concurrent use of System.in |
|
We have the same issue, with the same stack traces, so one is waiting for the System.in, and the others are waiting for that first one. Could the auxclasspath passed in the arguments file or in a separate file instead of the System.in? Blocking thread: 4 blocked threads: |
|
Could we fix the indication spotbugs maven plugin is thread-safe for now? |
|
I open a PR ins spotbugs to synchronize the System.in reads. We would need to synchronize the write also (else multiple writes would interleave in System.in). Alternativaly, we could add some shorter timeout in the read. It would not prevent issues, but it would prevent being stuck for 10 minutes. |
|
Not sure I follow how we are using UI during normal spotbugs run, that logic would indicate asking for input but that run isn't going to do that. feels like the GUI is being used during that type of run, can you confirm that? Having a hard time understanding it otherwise. The change on spotbugs is ok but it really shouldn't call out its due to maven plugin because gradle could do the same. |
|
@blacelle Unlikely I'll add this is not thread safe here as I don't plan to release until next spotbugs at the current moment. Can you show me full setup you are using though as I'm still perplexed at why system.in is used during normal spotbugs execution. That is for the GUI based on naming alone and system in expects to provide some system input. I'm confused how that would be getting executed. I haven't looked very deep though but just confused on how maven plugin run would ever be wanting system input like that. Possibly we are missing some features here, not sure or maybe your run was GUI based to launch that and that causes the issue. |
No.
I may dig into spotbugs-gradle if it was necessary. Here is a typical configuration producing this issue: https://github.com/solven-eu/cleanthat/blob/master/pom.xml#L517-L547 and I get the issue with cmd like ' |
|
Here is the mvn code triggering the ant task triggering FindBugs2 (as visible in the stack):
|
|
Maybe the issue is the empty |
|
I'm opening a PR to suggest removing this row. While opening the project in Eclipse, I get a stackoverflow: I also see: |
|
Check m2e version, it's been horribly bad for many months. Make sure you run off their snapshot line as it more stable. Currently should be 2.2.x.
Sent from my Verizon, Samsung Galaxy smartphone
Get Outlook for Android<https://aka.ms/AAb9ysg>
…________________________________
From: Benoit Lacelle ***@***.***>
Sent: Friday, January 27, 2023 2:40:13 AM
To: spotbugs/spotbugs-maven-plugin ***@***.***>
Cc: Jeremy Landis ***@***.***>; Comment ***@***.***>
Subject: Re: [spotbugs/spotbugs-maven-plugin] Parallel build without forking in a multi-module leads to spotbugs getting stuck (Issue #488)
I'm opening a PR to suggest removing this row. While opening the project in Eclipse, I get a stackoverflow:
java.lang.StackOverflowError
at org.apache.maven.plugin.PluginParameterExpressionEvaluator.evaluate(PluginParameterExpressionEvaluator.java:406)
at org.apache.maven.plugin.PluginParameterExpressionEvaluator.evaluate(PluginParameterExpressionEvaluator.java:145)
at org.apache.maven.plugin.PluginParameterExpressionEvaluator.evaluate(PluginParameterExpressionEvaluator.java:432)
at org.apache.maven.plugin.PluginParameterExpressionEvaluator.evaluate(PluginParameterExpressionEvaluator.java:145)
at org.apache.maven.plugin.PluginParameterExpressionEvaluator.evaluate(PluginParameterExpressionEvaluator.java:432)
at org.apache.maven.plugin.PluginParameterExpressionEvaluator.evaluate(PluginParameterExpressionEvaluator.java:145)
at
...
eclipse.buildId=4.25.0.I20220831-1800
java.version=17.0.4.1
java.vendor=Eclipse Adoptium
BootLoader constants: OS=macosx, ARCH=x86_64, WS=cocoa, NL=en_FR
Framework arguments: -product org.eclipse.epp.package.java.product -keyring /Users/blacelle/.eclipse_keyring
Command-line arguments: -os macosx -ws cocoa -arch x86_64 -product org.eclipse.epp.package.java.product -keyring /Users/blacelle/.eclipse_keyring
I also see:
Project build error: Resolving expression: '${java.test.release.version}': Detected the following recursive expression cycle in 'java.test.release.version': [java.test.release.version, java.release.version] pom.xml /spotbugs-maven-plugin line 1 Maven pom Loading Problem
Project build error: Resolving expression: '${java.release.version}': Detected the following recursive expression cycle in 'java.release.version': [java.release.version, java.test.release.version] pom.xml /spotbugs-maven-plugin line 1 Maven pom Loading Problem
—
Reply to this email directly, view it on GitHub<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fspotbugs%2Fspotbugs-maven-plugin%2Fissues%2F488%23issuecomment-1406128560&data=05%7C01%7C%7Cd55894688df24333505108db0039ba40%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638104020180391910%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=WjoFdD9nETU%2Fyi2Z1N6qxjGQB4SINE2ujJxZ5WM6pNw%3D&reserved=0>, or unsubscribe<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAHODIYEHPOOQ2OLL7QU2Y3WUN3V3ANCNFSM6AAAAAAQMNEKU4&data=05%7C01%7C%7Cd55894688df24333505108db0039ba40%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638104020180391910%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=SUJTKFm6OSeQ7lP3NUPaHF8g%2BFVep1g21M5cancHhR8%3D&reserved=0>.
You are receiving this because you commented.Message ID: ***@***.***>
|
|
Fixed via #535. Will release as soon as I can figure out why I cannot upgrade groovy to 4.0.8/4.0.9 that I need... |
…File argument The plugin versions before 4.7.3.1 passed the auxiliary classpath via the standard input, which caused a deadlock during parallel builds with non-forked spotbugs. spotbugs#488 Although this was fixed by removing the -auxclasspathFromInput, this broke the classpath logic. The solution is to pass the classpath using a file.
|
This fix is available in spotbugs-maven-plugin 4.7.3.1 Interesting to see that as a User (i.e. just having a ref to Spotbugs in some project, not opening Spotbugs source in Eclipse), I now also get issues around: m2e is up to date: I'm keen to open a ticket if this is not specific to myself (e.g. having open spotbugs-maven-plugin in Eclipse previously) |
|
issue fixed now and will be released shortly. |
I consider a 50+ multi-module project, with 100k+ java files. When applying spotbugs in our projects, we spend roughly 1 second per project. OK
I'm on the latest versions:
If I run the process with parallelism (e.g.
mvn install -T 8), our build tends to timeout. I attach a jstack.bug_spotbugs.txt
The main elements are :
The issue seems to occur very frequently (if not always) in our case (as long as the parallelism is high enough).
The main elements of the threaddump:
The text was updated successfully, but these errors were encountered: