History is hogging web app memory

[vladak]

When thinking about #3243, I came to realize that the web app suffers from similar problem: in general whenever a history is requested, the History object representing the complete history of given file/directory is loaded into memory:

opengrok/opengrok-web/src/main/webapp/history.jsp

Lines 81 to 84 in 610d908

	File file = cfg.getResourceFile(primePath);
	History hist;
	try {
	hist = HistoryGuru.getInstance().getHistoryUI(file);

What's more, it is actually stored in the request:

opengrok/opengrok-web/src/main/webapp/history.jsp

Line 99 in 610d908

request.setAttribute("history.jsp-hist", hist);

and used for paging.

Even for smaller repositories such as the OpenGrok Git repository this creates significant memory pressure. Here's a graph representing heap stats from a web app where I displayed history for couple of directories (including the top level directory) for indexed OpenGrok Git repository:

The Eden space grew by bunch of gigabytes !

[vladak]

The way how the History object is stored in FileHistoryCache makes it hard to do something about this. Firstly, it is compressed, secondly it is XML serialized Java object (#3539). So it has to be read whole into memory and then dissected.

[vladak]

Related to #3539 and #4023

[vladak]

Once a scheme to iterate over history (for both repository method and history cache) without reading all of it in memory is in place, the HistoryReader used during indexing should be converted to this scheme as well.