JvmErgonomicsTests.testExtractValidHeapSizeNoOptionPresent fails on 7.4

[dakrone]

It fails with:

org.elasticsearch.tools.launchers.JvmErgonomicsTests > testExtractValidHeapSizeNoOptionPresent FAILED
    java.lang.AssertionError: 
    Expected: a value greater than <0L>
         but: <0L> was equal to <0L>
        at __randomizedtesting.SeedInfo.seed([22C99F519C4644E0:535C63ED1A8DEF9E]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.tools.launchers.JvmErgonomicsTests.testExtractValidHeapSizeNoOptionPresent(JvmErgonomicsTests.java:59)

What's strange is that this did not print any reproduction line, so I wasn't able to try the exact test.

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.4+multijob-windows-compatibility/os=windows-2019/64/consoleFull
https://gradle-enterprise.elastic.co/s/mm3dqwj56d4aa/console-log

[elasticmachine]

Pinging @elastic/es-core-infra

[dakrone]

Another failure:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.3+multijob-windows-compatibility/os=windows-2016/164/console
https://gradle-enterprise.elastic.co/s/biyd4wfkrhnns

[droberts195]

Every single Windows 2012r2 build on the 7.4 branch this month has failed with this same error: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.4+multijob-windows-compatibility/os=windows-2012-r2/

[droberts195]

Every single Windows 2012r2 build on the 7.4 branch

The fact that this hasn't been failing in 7.x doesn't mean it's not a problem in 7.x. The 7.x builds all fail because of the minio problem - see #42829 - and the test report shows that JvmErgonomicsTests has not run at all because of this.

There are no differences in either JvmErgonomics.java or JvmErgonomicsTests.java between 7.4 and 7.x so the chances are this is still a problem in 7.x too.

It's working in master though: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-windows-compatibility/os=windows-2012-r2/203/testReport/org.elasticsearch.tools.launchers/JvmErgonomicsTests/

[alpar-t]

As predicted, here's the failure on 7.x https://gradle-enterprise.elastic.co/s/k6qhteq27zplc

[williamrandolph]

It looks like this is a JDK8 issue. The default value for MaxHeapSize is physical RAM / 4. On large machines, these numbers can be larger than what fits into a 32-bit integer, but that's what JDK 8 tries to print, so max heap sizes will be truncated. Some common values get truncated to 0, unfortunately. You can test this out by supplying such values as arguments to the -Xmx parameter.

Using Powershell, the results come out like this:

PS C:\Users\william_brafford\elasticsearch> C:\Users\jenkins\.java\java8\bin\java.exe -XX:+PrintFlagsFinal -Xmx3g -version | select-string MaxHeapSize

    uintx MaxHeapSize                              := 3221225472                          {product}

That is the correct value, i.e., 3g => 3221225472 bytes. But:

PS C:\Users\william_brafford\elasticsearch> C:\Users\jenkins\.java\java8\bin\java.exe -XX:+PrintFlagsFinal -Xmx8g -version | select-string MaxHeapSize

    uintx MaxHeapSize                              := 0                                   {product}

That's not right! Or rather, (8 * 2^30) % (2^32) = 0. Any multiple of 4 GB will have this problem, since 4GB = 4 * 2^30 B = 2^32 B.

Unfortunately, our Windows tests are executing on instances with 96 GB RAM, meaning the 24GB max heap size is getting truncated to 0 in the output that we're parsing.

JDK 9 and beyond don't have this problem, and the master branch uses JDK 11 for the runtime, so that's why we aren't seeing this issue on master.

As for the fix — is there a way to make this test meaningful, given that there's a bug in the JDK 8 output? Also, is this a potential problem on Linux, or is it Windows-specific? I'll keep digging on this.

[williamrandolph]

The behavior turns out to be system-dependent — "uintx" holds the correct values on my Mac, but not on my Windows test machines.

The JDK bug report flagging this issue is here: https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8074459

Since the problem behavior is in the JDK and not in our code, I think I will put out a PR that skips this test only on affected systems.

[williamrandolph]

On second thought, this bug might have some consequences for our ergonomic settings on affected machines. I'll figure out what the worst case would be and loop back for discussion.

[williamrandolph]

Note that we closed a related ticket by muting the tests on Windows: #44669 (comment)

[williamrandolph]

Here's what I've found. Two properties depend on the parsed value of MaxHeapSize:

• When io.netty.allocator.type is unset, JvmErgonomics sets it to unpooled when MaxHeapSize is less than or equal to 1g and to pooled when MaxHeapSize is greater than 1g. Thus, when this bug is present and the value is not defined, we'll get the wrong io.netty.allocator.type setting when physical memory is between N * 16g and (N * 16g) + 4g, for some natural number N.
• When MaxDirectMemorySize is unset, JvmErgonomics sets it to MaxHeapSize / 2. On affected systems, this means we will get an incorrect setting for MaxDirectMemorySize when the value is not defined and physical memory is greater than 16g.

In both cases, the workaround is for the user to define the value or upgrade to a JDK more recent than 8. Since we have workarounds, my main question is how to put this information in some place where support or our users can find it.

[alpar-t]

Would it be too harsh to bail telling the user about the bug and advising a heap size that doesn't fit the problematic pattern ? If my understanding is correct, just adding a byte to the heap would fix this. That way we wouldn't create any burden on users and support finding it at a slight inconvenience when everything lines up for the bug to reproduce, but would also save some trouble.

[williamrandolph]

A PR has been merged that removes the ergonomic io.netty.allocator.type setting: #48310

[williamrandolph]

This issue has been fixed by #48365 and #48657.