Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update our huge pages support #5678

Open
solardiz opened this issue Feb 20, 2025 · 1 comment
Open

Update our huge pages support #5678

solardiz opened this issue Feb 20, 2025 · 1 comment

Comments

@solardiz
Copy link
Member

  1. We have two instances of the code - one in yescrypt/yescrypt-platform.c, the other in memory.c. The latter is based on older yescrypt code and doesn't yet have this upstream yescrypt fix:
commit 1554b087ffe2d8c5c07c2795fe5cc21558b8a036
Author: Solar Designer <solar@openwall.com>
Date:   Thu Nov 17 16:45:03 2022 +0100

    When explicitly using huge pages, request the 2 MiB page size
    
    This should fix the issue (reported by Alex Forencich against libxcrypt)
    where on a system configured to use 1 GiB huge pages we'd fail on munmap()
    as we're only rounding the size up to a multiple of 2 MiB.  With the fix,
    we wouldn't use huge pages on such a system.
    
    Unfortunately, now we also wouldn't use huge pages on Linux kernels too old
    to have MAP_HUGE_2MB.
  1. We may want to revise our huge page thresholds. Apparently, lowering to 2 MiB helps Monero (see Monero format issues and room for improvement #5668). And with our different-size test vectors, the self-test may result in usage of huge pages (also remaining for benchmark/cracking) even if the actual (benchmark/loaded) hashes would be below threshold (this happens for Argon2, where we have 16 MiB test vectors, but benchmark at 4 MiB).

  2. We may want to report on huge page (non-)use, maybe at some verbosity level? Since a sysctl is usually needed for explicit huge pages, it's easy even for an experienced user to miss or forget this optimization and remain unaware of that.

@solardiz solardiz added this to the Definitely 2.0.0 milestone Feb 20, 2025
@solardiz
Copy link
Member Author

Unfortunately, now we also wouldn't use huge pages on Linux kernels too old to have MAP_HUGE_2MB.

I wonder if for JtR we want a different failure mode - fall back to the current behavior where we'd risk failing on munmap in the rare case of an old-kernel system (RHEL6 alike) configured to use 1 GiB huge pages.

solardiz added a commit to solardiz/john that referenced this issue Feb 28, 2025
to 2 MiB for most relevant formats, and 12 MiB for those using scrypt.
2 MiB threshold is important for Monero slow hash;
Between 8 and 16 MiB was previously found good for yescrypt without ROM
(upstream has it at 32 MiB to have RAM and ROM use different TLBs).

See openwall#5668, openwall#5678
solardiz added a commit that referenced this issue Feb 28, 2025
to 2 MiB for most relevant formats, and 12 MiB for those using scrypt.
2 MiB threshold is important for Monero slow hash;
Between 8 and 16 MiB was previously found good for yescrypt without ROM
(upstream has it at 32 MiB to have RAM and ROM use different TLBs).

See #5668, #5678
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant