Resource leak when using Maven plugin #559
Comments
|
Interesting, thanks for the detailed profiling! It should be the case that the classloaders are cached across projects, so there should only be a few created across the entire build. You get a big speedup from the JIT by reusing these classloaders as much as you can - for the gradle plugin we only ever clear them when someone runs
That is really surprising to me!! To my knowledge we don't manually spawn a single thread. My best guess at what's happening is:
Independent of that, I'm also suspicious that if a maven plugin declares a I know it's been a long time since you worked on this, but any thoughts @lutovich? |
Maven will create different classloaders and reuse them depending on the (plugin) configuration used in Maven projects. In parallel Maven builds ( |
|
Hey! Yeah, it feels like with parallel builds and multiple classloaders |
Thanks alot. Since it took us several month to get to the point where we are now, a week or two doesn't matter. We can always apply the formatter to single projects only which is not a problem. |
|
Seems like this problem is caused by the way Running spotless with Google code formatter works fine and the problem manifests once I re-add at least one Eclipse-based formatter. Here is my investigation branch https://github.com/lutovich/spotless/tree/fix-spotless-cache-leak with some printouts and a hack that makes Eclipse Java formatter step pass when building openhab-addons. The hack is a very dirty one and a real fix would require some more effort. An idea for a fix: instead of using the "physical" settings files for cache keys use "logical" files. For example, every cache key for Eclipse Java format step would use the configured Any ideas or thoughts about this are greatly appreciated! |
|
Aha! We had a similar problem (very different symptom though) when adding tsfmt support to maven-spotless (#553). The problem there is that I see now in my notes that I left myself a todo, which was:
But it never got to the top of my todo :). So whaddya think - can we replace every call to |
|
Spotbugs and Checkstyle Maven plugins seem to also use I remember looking at them for inspiration :) Do you think it is possible to use actual file content in |
|
Added another commit to that branch to handle hashing for files and directories differently: diff. Input is much appreciated! |
|
We can definitely change With that in mind, a few points on the diff:
|
|
Making Maven plugin use human-readable names for output files would be nice. I'm not sure how to do this nicely though. Creating a name for the output file is easy when a step is configured with a local file: <eclipse>
<file>${basedir}/eclipse-fmt.xml</file>
</eclipse>then we could just name the output file However, it could be a bit tricky when a step is configured with a URL: <eclipse>
<file>http://my-site.com/eclipse-fmt.xml</file>
</eclipse>where URLs might have redirects or query params. I might be overthinking it, any ideas are appreciated! Please take a look at the attached PR. It makes output file names predictable but not human-readable. Thus it is possible to sort on file names. This PR makes it possible to run |
|
We still don't have a root fix for you. However, we do have an unfinished PR which would apply Spotless only to files which have changed since some git reference (e.g. |
|
We just published |
|
That‘s good news. I‘ll check in the next days. Thanks for your support. |
|
Also, @lutovich did all the work to make this happen, I just pressed the "release" button. |
|
Unfortunately it doesn't work. If I run The interesting thing is: If i change to |
|
@J-N-K sorry about this! Unfortunately, I do not have access to a Windows machine at the moment. You are right, the code looks like it should replace all separators with Unix-style separators. Do you think you could try to debug this a bit with Maybe you could in one shell run: and in another shell run: I tried this debugging locally on a different line in |
|
Would be very interesting to have that debug info. I just pushed up a possible solution in #639, which exposes a surprising test failure. We run our unit tests on windows & unix, but it appears there was a windows-only line-endings regression in 2.0 that went untested. I will investigate... |
|
I didn't manage to use jdb from the command line, but used the IntelliJ remote debugging instead.
Additionally I checked that on the first project in that build (the previous one is 6/277) and there
I digged a little bit deeper. And if I disable the karaf-maven-plugin, everything works. So it seems that either our configuration needs to be adapted or that plugin changes the line-ending property of |
|
We've released a fix in |
Recently we started using spotless in openHAB (http://github.com/openhab/openhab-addons). We are currently not able to do a full build (around 260 sub-projects) with checks enabled due to resource issues. It seems that this does not depend on the OS as it shows on our Unix-based CI and also on local builds under Windows.
A first analysis can be found here: openhab/openhab-addons#7449 (comment). Looking at a heap dump it seems that the
SpotlessCacheis instantiated with a newFeatureClassloadereach time a check is called but never cleared.The pom including the configuration can be found here: https://github.com/openhab/openhab-addons/blob/2.5.x/pom.xml
The text was updated successfully, but these errors were encountered: