[MRRESOURCES-106] ProcessRemoteResourcesMojo#getProjects() can end up consuming a lot of memory for big projects

[jira-importer]

Thomas Mortagne opened MRRESOURCES-106 and commented

At XWiki we are using the remote resource plugin to generate a NOTICE files we put in the META-INF of all jars. The file can be found on https://github.com/xwiki/xwiki-commons/blob/master/xwiki-commons-tools/xwiki-commons-tool-license-resources/src/main/resources/META-INF/NOTICE.vm.

We have a lot of memory issues and we kept increasing the max memory but it was starting to be a bit insane (we now get a OOM with -Xmx2500m) so I finally took a profiler to try to figure out where all this memory goes.

We did noticed for a while that remote resource plugin is taking longer and longer to execute during the build so I had my doubts already.
What Yourkit is telling me is that almost half of the memory (400MB here because I reduced the max memory for it to fail earlier) is retained by an ArrayList of MavenProject instances located in ProcessRemoteResourcesMojo#getProjects().

It can be reproduced by building https://github.com/xwiki/xwiki-platform (you will need to add some repositories in your settings.xml, you can find them on http://dev.xwiki.org/xwiki/bin/view/Community/Building/#HInstallingMaven).

Also I can probably put the memory dump I have somewhere if someone wants to download it.

---

Affects: 1.5

Issue Links:

• MRRESOURCES-94 Performance issue in ProcessRemoteResourcesMojo.configureVelocityContext(...)

[jira-importer]

Hervé Boutemy commented

fixed in MRRESOURCES-94

[jira-importer]

Falko Modler commented

Hervé Boutemy I don't think my fix for MRRESOURCES-94 is going to help here:
The OP is actually using a velocity template which does use projectsSortedByOrganization, so the (expensive) lookup will still kick in, only at a later point.

[jira-importer]

Hervé Boutemy commented

ok, reopening and changing the "duplicates" link to "relates to"

do you know why getProjects() can be so resource intensive?

[jira-importer]

Thomas Mortagne commented

getProjects() can be so resource intensive

Because MavenProject instances are unfortunately quite resource intensive. Not sure there is an easy fix for this.

What I did for XWiki is to introduce a way indicate what exactly will each MavenProject contains: no projects info at all (in many use cases you don't actually care about that and this is the default in the mojo I wrote), full (current behavior of the remote resource plugin), only license and minimal project coordinates (our use case), etc. See https://github.com/xwiki/xwiki-commons/blob/master/xwiki-commons-tools/xwiki-commons-tool-remote-resource-plugin/src/main/java/org/xwiki/tool/resource/XWikiProcessRemoteResourcesMojo.java.

[jira-importer]

Hervé Boutemy commented

oh, I just understood that "getProjects()" is not "get projects from reactor" but "get current project dependencies"...
in fact, I don't get why we get these dependencies as MavenProject, since we won't build them: probably org.apache.maven.model.Model would be less costly

[jira-importer]

Thomas Mortagne commented

From what I remember most of the MavenProject size is located in Model so not sure it will make a huge difference.

[jira-importer]

Hervé Boutemy commented

effective model is built to do the dependency resolution: I don't really get what is the processing to transform an effective model to a MavenProject, but I suppose this is what costs a lot of resources...