Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crypto modern api #18

Merged
merged 9 commits into from
Feb 20, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .clang-format
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
IndentWidth: 4
ColumnLimit: 100
133 changes: 89 additions & 44 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ on:
workflow_dispatch:

jobs:
test:

Ubuntu:
name: OTP ${{matrix.otp}} / rebar3 ${{matrix.rebar3}} / OS ${{matrix.os}}
strategy:
fail-fast: false
Expand All @@ -20,44 +21,13 @@ jobs:
- otp: '24'
rebar3: '3.22.1'
os: 'ubuntu-22.04'
- otp: '27'
rebar3: '3.24.0'
os: 'windows-2022'
- otp: '27'
rebar3: '3.24.0'
os: 'macos-latest'
runs-on: ${{matrix.os}}
steps:
- uses: actions/checkout@v4

# OS setups
- name: Ubuntu/Windows – Prepare Erlang
uses: erlef/setup-beam@v1
- uses: erlef/setup-beam@v1
with:
otp-version: ${{matrix.otp}}
rebar3-version: ${{matrix.rebar3}}
if: ${{ matrix.os != 'macos-latest' }}
- name: Windows - Enable Developer Command Prompt
uses: ilammy/msvc-dev-cmd@v1
if: ${{ matrix.os == 'windows-2022' }}
- name: Windows - Install openssl
shell: pwsh
run: |
choco install openssl
echo "OPENSSL_INSTALL_DIR=""C:\Program Files\OpenSSL""" >> $env:GITHUB_ENV
if: ${{ matrix.os == 'windows-2022' }}
- name: MacOS – Prepare Brew
run: |
brew --version
brew cleanup --prune=all -s
brew autoremove
brew untap homebrew/cask homebrew/core
brew update
if: ${{ matrix.os == 'macos-latest' }}
- name: MacOS - Prepare Erlang
run: brew install erlang rebar3
if: ${{ matrix.os == 'macos-latest' }}

# caches
- name: Restore _build
uses: actions/cache@v4
Expand All @@ -69,7 +39,6 @@ jobs:
with:
path: ~/.cache/rebar3
key: rebar3-cache-for-os-${{matrix.os}}-otp-${{matrix.otp}}-rebar3-${{matrix.rebar3}}-hash-${{hashFiles('rebar.lock')}}

# tests
- run: rebar3 as test get-deps
- run: rebar3 as test compile
Expand All @@ -87,13 +56,51 @@ jobs:
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

test-on-s390x:
MacOS:
name: OTP ${{matrix.otp}} / rebar3 ${{matrix.rebar3}} / OS ${{matrix.os}}
strategy:
fail-fast: false
matrix:
os: ['macos-14', 'macos-15']
otp: ['27']
rebar3: ['3.24.0']
runs-on: ${{matrix.os}}
steps:
- uses: actions/checkout@v4
# OS setups
- name: Prepare Brew
run: |
brew --version
brew cleanup --prune=all -s
brew autoremove
brew untap homebrew/cask homebrew/core
brew update
- run: brew install erlang rebar3
# caches
- name: Restore _build
uses: actions/cache@v4
with:
path: _build
key: _build-cache-for-os-${{matrix.os}}-otp-${{matrix.otp}}-rebar3-${{matrix.rebar3}}-hash-${{hashFiles('rebar.lock')}}
- name: Restore rebar3's cache
uses: actions/cache@v4
with:
path: ~/.cache/rebar3
key: rebar3-cache-for-os-${{matrix.os}}-otp-${{matrix.otp}}-rebar3-${{matrix.rebar3}}-hash-${{hashFiles('rebar.lock')}}
# tests
- run: rebar3 as test get-deps
- run: >
CFLAGS="-I/opt/homebrew/opt/openssl/include"
LDFLAGS="-L/opt/homebrew/opt/openssl/lib"
rebar3 as test compile
- run: rebar3 as test ct

s390x:
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v4
- name: Setup emulator
run: |
sudo docker run --rm --privileged tonistiigi/binfmt:qemu-v9.2.0
run: sudo docker run --rm --privileged tonistiigi/binfmt:qemu-v9.2.0
- name: Run build
uses: uraimo/run-on-arch-action@v2.8.1
with:
Expand All @@ -102,10 +109,48 @@ jobs:
install: |
apt-get update -y
DEBIAN_FRONTEND=noninteractive apt-get install -y rebar3 gcc libssl-dev
run: |
echo "---rebar3 as test get-deps---"
rebar3 as test get-deps
echo "---rebar3 as test compile---"
rebar3 as test compile
echo "---rebar3 as test ct---"
rebar3 as test ct
run: rebar3 as test do get-deps, compile, ct

# Windows:
# name: OTP ${{matrix.otp}} / rebar3 ${{matrix.rebar3}} / OS ${{matrix.os}}
# strategy:
# fail-fast: false
# matrix:
# os: ['windows-2022']
# otp: ['27']
# rebar3: ['3.24.0']
# runs-on: ${{matrix.os}}
# steps:
# - uses: actions/checkout@v4
# - uses: erlef/setup-beam@v1
# with:
# otp-version: ${{matrix.otp}}
# rebar3-version: ${{matrix.rebar3}}
# - name: Windows - Enable Developer Command Prompt
# uses: ilammy/msvc-dev-cmd@v1
# if: ${{ matrix.os == 'windows-2022' }}
# - name: Windows - Install openssl
# shell: pwsh
# run: |
# choco install openssl
# echo "OPENSSL_INSTALL_DIR=""C:\Program Files\OpenSSL""" >> $env:GITHUB_ENV
# if: ${{ matrix.os == 'windows-2022' }}
# # caches
# - name: Restore _build
# uses: actions/cache@v4
# with:
# path: _build
# key: _build-cache-for-os-${{matrix.os}}-otp-${{matrix.otp}}-rebar3-${{matrix.rebar3}}-hash-${{hashFiles('rebar.lock')}}
# - name: Restore rebar3's cache
# uses: actions/cache@v4
# with:
# path: ~/.cache/rebar3
# key: rebar3-cache-for-os-${{matrix.os}}-otp-${{matrix.otp}}-rebar3-${{matrix.rebar3}}-hash-${{hashFiles('rebar.lock')}}
# # tests
# - run: dir "C:\Program Files\OpenSSL\"
# - run: rebar3 as test get-deps
# - run: >
# set CFLAGS="/I${OPENSSL_INSTALL_DIR}/include" &
# set LDFLAGS="/LIBPATH:${OPENSSL_INSTALL_DIR}/lib" &
# rebar3 as test compile
# - run: rebar3 as test ct
21 changes: 11 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,20 +1,21 @@
# Fast PBKDF2

[![Actions Status](https://github.com/esl/fast_pbkdf2/workflows/ci/badge.svg)](https://github.com/esl/fast_pbkdf2/actions)
[![codecov](https://codecov.io/gh/esl/fast_pbkdf2/branch/main/graph/badge.svg)](https://codecov.io/gh/esl/fast_pbkdf2)
[![Hex](http://img.shields.io/hexpm/v/fast_pbkdf2.svg)](https://hex.pm/packages/fast_pbkdf2)
[![Hex pm](https://img.shields.io/hexpm/v/fast_pbkdf2.svg)](https://hex.pm/packages/fast_pbkdf2)
[![Hex Docs](https://img.shields.io/badge/hex-docs-lightgreen.svg)](https://hexdocs.pm/fast_pbkdf2/)
[![Downloads](https://img.shields.io/hexpm/dt/fast_pbkdf2.svg)](https://hex.pm/packages/fast_pbkdf2)
[![GitHub Actions](https://github.com/esl/fast_pbkdf2/workflows/ci/badge.svg?branch=main)](https://github.com/esl/fast_pbkdf2/actions?query=workflow%3Aci+branch%3Amain)
[![Codecov](https://codecov.io/gh/esl/fast_pbkdf2/branch/main/graph/badge.svg)](https://codecov.io/gh/esl/fast_pbkdf2)
[![License](https://img.shields.io/hexpm/l/fast_pbkdf2.svg)](https://github.com/esl/fast_pbkdf2/blob/main/LICENSE)

`fast_pbkdf2` is an Erlang implementation of [PBKDF2][PBKDF2], where the algorithm is a carefully-optimised NIF that uses timeslicing and nif scheduling to respect the latency properties of the BEAM.
All OTP versions from OTP18 have been tested manually and should work correctly, but on CI we support only from 21.3

## Building
`fast_pbkdf2` is a rebar3-compatible OTP application, that uses the [port_compiler](https://github.com/blt/port_compiler) for the C part of the code.

Building is as easy as `rebar3 compile`, and using it in your projects as
```erlang
{deps,
[{fast_pbkdf2, "1.0.0"}]}.
{plugins, [pc, rebar3_hex]}.
[{fast_pbkdf2, "~> 2.0"}]}.
{provider_hooks,
[{pre,
[{compile, {pc, compile}},
Expand All @@ -28,7 +29,7 @@ DerivedPassword = fast_pbkdf2:pbkdf2(Hash, Password, Salt, IterationCount)
```
where `Hash` is the underlying hash function chosen as described by
```erlang
-type sha_type() :: crypto:sha1() | crypto:sha2().
-type sha_type() :: crypto:sha1() | crypto:sha2() | crypto:sha3().
```

### Custom `dkLen`
Expand All @@ -43,16 +44,16 @@ PBKDF2 is a challenge derivation method, that is, it forces the client to comput
Is partial. We don't expect to have the fastest implementation, as that would be purely C code on GPUs, so unfortunately an attacker will pretty much always have better chances there. _But_ we can make the computation cheap enough for us that other computations —like the load of a session establishment— will be more relevant than that of the challenge; and also that other defence mechanisms like IP blacklisting or traffic shaping, will fire in good time.

### The outcome
On average it's 10x faster on the machines I've tested it (you can compare using the provided module in `./benchmarks/bench.ex`), but while the erlang implementation consumes memory linearly to the iteration count (1M it count with 120 clients quickly allocated 7GB of RAM, and 1M is common for password managers for example), the NIF implementation does not allocate any more memory. Also, the NIFS spend all of their time in user level alone, while the erlang one jumps to system calls in around ~2% of the time (I'd guess due to some heavy allocation and garbage collection patterns).
On average it's 30% faster than the pure OpenSSL implementation, which `crypto:pbkdf2_hmac/5` calls without yielding, and 10x times faster (and x3N less memory, where N is the iteration count!) than a pure erlang equivalent (you can compare using the provided module in `./benchmarks/bench.ex`).

## Credit where credit is due
The initial algorithm and optimisations were taken from Joseph Birr-Pixton's
[fastpbkdf2](https://github.com/ctz/fastpbkdf2)'s repository.

## Read more:
* Password-Based Cryptography Specification (PBKDF2): [RFC8018](https://tools.ietf.org/html/rfc8018#section-5.2)
* HMAC: [RFC2104]( https://tools.ietf.org/html/rfc2104)
* SHAs and HMAC-SHA: [RFC6234](https://tools.ietf.org/html/rfc6234)
* HMAC: [RFC2104](https://datatracker.ietf.org/doc/html/rfc2104)
* SHAs and HMAC-SHA: [RFC6234](https://datatracker.ietf.org/doc/html/rfc6234)

[MIM]: https://github.com/esl/MongooseIM
[PBKDF2]: https://tools.ietf.org/html/rfc8018#section-5.2
Loading
Loading