Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix failures to allocate arrays with a very large Java heap size #1986

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

EthanNelson-Moore
Copy link

@EthanNelson-Moore EthanNelson-Moore commented Dec 17, 2024

Description

The maximum length of a Java array is not exactly Integer.MAX_VALUE, but slightly less due to the space taken up by the object header. The exact maximum differs depending on the platform and Java version. This was already accounted for in one instance, but not others. This was causing MarkDuplicates to fail with a Java heap size of slightly less than 1TB when MAX_RECORDS_IN_RAM was not specified (and therefore calculated based on the amount of available memory). This commit fixes the other instances and changes the maximum size in the existing instance to Integer.MAX_VALUE - 32 instead of Integer.MAX_VALUE - 5 to decrease the likelihood of allocation failures on different Java versions and platforms.


Checklist (never delete this)

Never delete this, it is our record that procedure was followed. If you find that for whatever reason one of the checklist points doesn't apply to your PR, you can leave it unchecked but please add an explanation below.

Content

  • Added or modified tests to cover changes and any new functionality - It would be hard to automatically test this change since triggering the issue requires an extremely large amount of memory, which is not likely to be the case in a CI pipeline.
  • Edited the README / documentation (if applicable) - This is a bug fix, so there is no need to update the README or documentation, only the changelog of a future release.
  • All tests passing on github actions

Review

  • Final thumbs-up from reviewer
  • Rebase, squash and reword as applicable

For more detailed guidelines, see https://github.com/broadinstitute/picard/wiki/Guidelines-for-pull-requests

The maximum length of a Java array is not exactly Integer.MAX_VALUE,
but slightly less due to the space taken up by the object header.
The exact maximum differs depending on the platform and Java
version. This was already accounted for in one instance, but not
others. This commit fixes the other instances and changes the
maximum size in the existing instance to Integer.MAX_VALUE - 32
instead of Integer.MAX_VALUE - 5 to decrease the likelihood of
allocation failures on different Java versions and platforms.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant