fix(reports): tune flags for reliability

[andrewazores]

Fixes #697

Related cryostatio/cryostat-operator#277

Image available at quay.io/andrewazores/cryostat:serialgc-report

[andrewazores]

recordings.zip

This .zip contains two recording files I collected from the smoketest sample apps, both using the Profiling template while the applications sat idle for some time - one twice as long as the other, so the recording is twice the size. Unzip it, open a terminal, and navigate to where the two recordings are after unzipping.

$ CRYOSTAT_IMAGE=quay.io/cryostat/cryostat:latest sh smoketest.sh
# in a separate terminal:
$ curl -vk -F recording=@jmx-listener-55d48f7cfc-p9st4_profiling_20211004T222522Z.jfr https://0.0.0.0:8181/api/v1/recordings
$ curl -vk -F recording=@quarkus-test-6bd55b764c-vnq4l_profiling_20211004T222527Z.jfr https://0.0.0.0:8181/api/v1/recordings
# this request should succeed
$ curl -vk https://0.0.0.0:8181/api/v1/reports/jmx-listener-55d48f7cfc-p9st4_profiling_20211004T222522Z.jfr
# this request is likely to fail
$ curl -vk https://0.0.0.0:8181/api/v1/reports/quarkus-test-6bd55b764c-vnq4l_profiling_20211004T222527Z.jfr

Repeat the same test, but with CRYOSTAT_IMAGE=quay.io/andrewazores/cryostat:serialgc-report and both of the report requests should succeed. So, the flag tuning in this PR does seem to improve the ability for the report generation to handle larger recordings, although I have yet to determine the new ceiling with these changes. It may also depend on the event template in use/which events are recorded, since the JMC automated rules analysis memory usage profile probably changes somewhat depending on which rules are being evaluated and how much work they each have to do.

[ebaron]

I was able to generate a report with no problems for Cryostat itself, when deployed in OpenShift. 30 seconds, Profiling template.

[andrewazores]

Yea, I tried that yesterday and had no problem as well. Although with the profiling template, the maximum recording size that can be processed is pretty small - it's evidently somewhere between the two recordings I have in my last comment.

FWIW I still haven't been able to observe any noticeable (by "eye") difference in report generation whether the target is Cryostat itself or some external JVM.

[ebaron]

I tried the same, 30s Profiling, without this change and it runs out of memory. JFR file size is 615KB.

[ebaron]

Looks good!