Using Corretto in Amazon ECS with 1024 CPU units

[TheTweak]

Container CPU units in ECS are mapped to CPU shares in Docker. 1024 CPU shares are a default value in Docker, and so Corretto (as well as plain OpenJDK), during active processor count, treats it as if no shares are set and returns all system processors count.

In ECS, 1024 CPU units is not a default value, and so user might expect that his JVM will see only one core in processor count, not all system processors count.

We overcome this by setting -XX:ActiveProcessorCount=1 explicitly when 1024 CPU units are set in ECS.

I think that making Corretto aware of cloud environment and change default behaviour if it runs in ECS is not a solution, but may be a special flag to JVM that does change it, is something more appropriate?

[ziyiluo]

@TheTweak thanks for reporting this! This is a very interesting edge case. 1024 is just a default number in docker but it is used in JDK as "PER_CPU_SHARES" to calculate the total number of CPU cores. However there's also this line that recognizes 1024 as "no share setup" therefore returns all system available cores. We have found the root cause but we are still discussing if this should be a JDK fix or ECS fix.

Will let you know whenever we have an update.

[ziyiluo]

Hi @TheTweak Thanks for waiting.

We reached out to the ECS team regarding your problem and they confirmed that this is a known issue. Quote from ECS's response: "ECS, Docker/Linux, and Java all have different ideas about what a cpu share means conceptually. ECS treats it as a reservable block of cpu power while Java treats this as a means to guess how many threads it should run." ECS believes that recognizing 1024 CPU shares differently is not an acceptable semantics change. We also learned from ECS that one noticable difference between ECS and docker is that the default CPU shares is 2 rather than 1024. See amazon-ecs-agent#1735 for more information.

The behavior of the container awareness in JDK has been discussed in the OpenJDK community a few years ago and ignoring the cpu shares when the value is default is an expected behavior.

Therefore the issue is not fixable in either ECS or JDK because we cannot break the existing behavior. As a not so elegant workaround, since CPU-shares is a soft limitation in docker, you may set the value to 1023 in ECS to let JVM recognize your CPU configuration.

[TheTweak]

Got it, thanks for the details!