Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WASI-20. Low memory prover (512k possible). Native pthread support (15s wasm time). #434

Merged
merged 111 commits into from
Jun 8, 2023
Merged

WASI-20. Low memory prover (512k possible). Native pthread support (15s wasm time). #434

merged 111 commits into from
Jun 8, 2023

Conversation

charlielye
Copy link
Contributor

@charlielye charlielye commented May 6, 2023 •
edited
Loading

barretenberg

Wins:

  • Move to WASI 20.
  • Latest clang etc.
  • Proper static initialisation of globals.
  • Can remove OpenMP (for now can switch to new threading with -DOMP_MULTITHREADING=OFF).
  • Can remove asyncify (conditional variables + threads as alternative).
  • Less memory usage and copying memory.
  • Much faster performance, granular threading, (AC was very coarse).
  • Same threading model as native code.
  • Could get rid of worker queue (but don't yet, just disabled cbinds).
  • Get rid of host environment threading (PooledPippenger PooledFft) in ts code.
  • Unified bb.js browser/node code with very simple abstraction layer to deal with differences.
  • Same simple.rawtest.ts works in node and when compiled to browser via webpack.
  • No more node polyfills in bb.js.

How:

  • wasi "_initialize" solves re-entry crash, enables move to wasi 20.
  • wasi-libc bleeding edge fork for increased (not perfect) join stability.
  • parallel_for replaces OpenMP. (several versions, using "parallel_for_mutex_pool" to avoid stability bug).
  • SlabAllocator enables 512k gates within 4GB.
  • Beyond that, PolynomialStoreCache and host engine swapping (unused currently as it's kinda janky TBH).
  • Polynomial coeffs in shared_ptr + SlabAllocator.

bb.js

We are moving towards standalone prover/verifier binaries to be maintained by bb team(s), and to be consumed by noir team(s) (and whoever else).
This is the first implementation of such a prover/verifier.

Wins:

  • Completely architecture/os agnostic. As a wasm it runs everywhere, even the browser.
  • Handles up to 512k gates, proving time 18s on M1 Mac (builds off prior work).
  • New cleaned up AcirComposer interface that's simple and sane to use.

charlielye and others added 30 commits April 9, 2023 08:47
@charlielye
Rm foundation
40245cc
@charlielye
Remove foundation submodule, add build-system
807882e
@charlielye
Initial bindgen work.
c9922f1
@charlielye
Schnorr and blake2s binds/tests.
9ebd5e0
@charlielye
Tidy.
a1d0e9b
@charlielye
GPT4 created rust bindgen.
70d5c74
@charlielye
Improve.
e758776
@charlielye
Merge remote-tracking branch 'origin/master' into cl/ts
53e5df7
@charlielye
Broken initial
7bba67a
@charlielye
Almost.£
f8c4f85
@charlielye
Fix test after master merge.
74ef953
@charlielye
Debug build preset wasm-dbg for stack traces etc.
a3aca68
@charlielye
Async call support. Refactor binds. Common is module. WASM stubs unus…
aaa97da 
…ed wasi.
@charlielye
Move some c_bind to module root and make use new abi.
0ef69cd
@charlielye
Think things pass.
fee3263
@charlielye
Poly store returns value.
68055c2
@charlielye
Wasm build
8ca5058
@charlielye
Debugging...
efbc030
@charlielye
Debugging
8fc4eb8
@suyash67
Fix bootstrap errors.
5158866
@suyash67
hacky fix (still to get to the bottom of this).
2d9d65d
@charlielye
Fiddling with threads.
028ed58
@suyash67
Fix (other address should also be "unsigned", thanks to Charlie)
0fa9271
@charlielye
Merge branch 'cl/poly_store' into cl/wasm_pthread
d934377
@charlielye
Initial working pthreads.
1e4cbf2
@charlielye
Cond var example.
b3b7662
@charlielye
Threading work.
b15338a
@charlielye
Some cleanup/fixes
29c43e0
@charlielye
More threading.
930d8b6
@charlielye
Cheapen some polynomial copies. Remove more openmp.
3d4d636
charlielye and others added 24 commits June 5, 2023 11:27
@charlielye
bb.js improv.
14ac983
@charlielye
Bump
ed15f99
@charlielye
Merge remote-tracking branch 'origin/master' into cl/wasm_pthread
ea026aa
@charlielye
Merge branch 'cl/wasm_pthread_swap_mem_working' into cl/wasm_pthread
16b203b
@charlielye
Fix horrific nasty memory bug.
3a4fd7c
@charlielye
cbind stuff. size hint stuff.
c569502
@Rumata888
Out-of-scope fix
487b9c2
@Rumata888
Second fix
9c006d8
@Rumata888
Same fixes for standard and turbo
ef400d4
@vezenovm
rename tests
e1ea4e5
@charlielye
Lots of goodness. Initial attempt to test in ci.
b861429
@charlielye
ci
5034a96
@charlielye
Fix
91f92cf
@charlielye
header update.
0fd4310
@charlielye
Test readme
9eaee43
@codygunton
Init global crs from string.
b5393e3
@codygunton
Bump packages commit (circuits tests should pass)
5838174
@codygunton
Use global SRS in tests
1cdf4f5
@charlielye
Fix file crs wasm.
38d6e6e
@charlielye
Merge branch 'cl/wasm_pthread' of github.com:aztecprotocol/barretenbe…
53b8b06 
…rg into cl/wasm_pthread
@codygunton
Fix loc and get the srs
6d842f3
@charlielye
Fix.
8f38a00
@charlielye
Merge branch 'cl/wasm_pthread' of github.com:aztecprotocol/barretenbe…
b488545 
…rg into cl/wasm_pthread
@codygunton
Fix recursion tests
7608038
@codygunton
Copy link
Collaborator

Charlie gave me a tour of changes relevant to the cryptography team. Otherwise we are foregoing a review due to the massive scope, the amount of new code, the fact that things are very much in flux, and the fact that this is not intended for production.

@codygunton codygunton merged commit 205a3f0 into master Jun 8, 2023
@codygunton codygunton deleted the cl/wasm_pthread branch June 8, 2023 19:41
ludamad pushed a commit to AztecProtocol/aztec-packages that referenced this pull request Jul 22, 2023
@charlielye @suyash67
@vezenovm @Maddiaa0 @ledwards2225 @Rumata888 @kevaundray @zac-williamson @codygunton
WASI-20. Low memory prover (512k possible). Native pthread support (1…
1924a24 
…5s wasm time). (AztecProtocol/barretenberg#434)

* Remove foundation submodule, add build-system
* Initial bindgen work.
* GPT4 created rust bindgen.
* Debug build preset wasm-dbg for stack traces etc.
* Async call support. Refactor binds. Common is module. WASM stubs unused wasi.
* Move some c_bind to module root and make use new abi.
* Initial working pthreads.
* Polynomial cache
* Custom poly allocator. Prealloc.
* Cleanup. Several parallel_for solutions. Might be worth keeping them around until threading work stabilises.
* Improve mem usage by slab allocating large parts of pippenger runtime state, and allowing slab reuse by removing from pk.
* Don't need binaryen.
* Bump alpine to get clang 16
* Backwards compatible cbinds.
* Move slab allocator to common. Init slab allocator just before creating pk.
* Dont need bleeding edge fork of libc anymore
* bb.js is standalone prover that handles noir constraint/witness output. (AztecProtocol/barretenberg#471)
* format msgpack serialization and excldue msgpack-c from clang-format (AztecProtocol/barretenberg#467)
* Fix horrific nasty memory bug.

---------

Co-authored-by: Suyash Bagad <suyashnbagad1997@gmail.com>
Co-authored-by: Maxim Vezenov <mvezenov@gmail.com>
Co-authored-by: Maddiaa <47148561+Maddiaa0@users.noreply.github.com>
Co-authored-by: ledwards2225 <98505400+ledwards2225@users.noreply.github.com>
Co-authored-by: Innokentii Sennovskii <isennovskiy@gmail.com>
Co-authored-by: kevaundray <kevtheappdev@gmail.com>
Co-authored-by: zac-williamson <blorktronics@gmail.com>
Co-authored-by: codygunton <codygunton@gmail.com>
ludamad pushed a commit to AztecProtocol/aztec-packages that referenced this pull request Jul 24, 2023
@charlielye @suyash67
@vezenovm @Maddiaa0 @ledwards2225 @Rumata888 @kevaundray @zac-williamson @codygunton
WASI-20. Low memory prover (512k possible). Native pthread support (1…
f8532d6 
…5s wasm time). (AztecProtocol/barretenberg#434)

* Remove foundation submodule, add build-system
* Initial bindgen work.
* GPT4 created rust bindgen.
* Debug build preset wasm-dbg for stack traces etc.
* Async call support. Refactor binds. Common is module. WASM stubs unused wasi.
* Move some c_bind to module root and make use new abi.
* Initial working pthreads.
* Polynomial cache
* Custom poly allocator. Prealloc.
* Cleanup. Several parallel_for solutions. Might be worth keeping them around until threading work stabilises.
* Improve mem usage by slab allocating large parts of pippenger runtime state, and allowing slab reuse by removing from pk.
* Don't need binaryen.
* Bump alpine to get clang 16
* Backwards compatible cbinds.
* Move slab allocator to common. Init slab allocator just before creating pk.
* Dont need bleeding edge fork of libc anymore
* bb.js is standalone prover that handles noir constraint/witness output. (AztecProtocol/barretenberg#471)
* format msgpack serialization and excldue msgpack-c from clang-format (AztecProtocol/barretenberg#467)
* Fix horrific nasty memory bug.

---------

Co-authored-by: Suyash Bagad <suyashnbagad1997@gmail.com>
Co-authored-by: Maxim Vezenov <mvezenov@gmail.com>
Co-authored-by: Maddiaa <47148561+Maddiaa0@users.noreply.github.com>
Co-authored-by: ledwards2225 <98505400+ledwards2225@users.noreply.github.com>
Co-authored-by: Innokentii Sennovskii <isennovskiy@gmail.com>
Co-authored-by: kevaundray <kevtheappdev@gmail.com>
Co-authored-by: zac-williamson <blorktronics@gmail.com>
Co-authored-by: codygunton <codygunton@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@charlielye charlielye

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@charlielye @codygunton @suyash67 @Rumata888 @vezenovm