Test generators for kzg-4844 libraries #3274

dankrad · 2023-02-25T22:30:14Z

Based on #3272.

Tests to be written for

hwwhww · 2023-02-26T03:41:19Z

Great work!

Some installation errors:

Linux (CircleCI)

Operating System: Ubuntu 20.04.5 LTS
OSType: linux
Architecture: x86_64
Python: CPython 3.8

Collecting py_arkworks_bls12381==0.3.1
  Downloading py_arkworks_bls12381-0.3.1.tar.gz (11 kB)
  Installing build dependencies ... -� �\� �|� �/� �done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... -� �error
    ERROR: Command errored out with exit status 1:
     command: /home/circleci/specs-repo/venv/bin/python3 /home/circleci/specs-repo/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpomv4ghkj
         cwd: /tmp/pip-install-3wfzp7my/py-arkworks-bls12381_1a72aa3f75004b0cad637ad55afd6093
    Complete output (6 lines):
    
    Cargo, the Rust package manager, is not installed or is not on PATH.
    This package requires Rust and Cargo to compile extensions. Install it through
    the system's package manager or via https://rustup.rs/
    
    Checking for Rust toolchain....
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/80/36/e237fbcf36960097892686f34414cef260132f04299619e7e6d4d0f4858b/py_arkworks_bls12381-0.3.1.tar.gz#sha256=6eb0f01d4545747626753fe5baf4c2a1ff014b03a7eb6531a6941ee5e6d5bce6 (from https://pypi.org/simple/py-arkworks-bls12381/) (requires-python:>=3.7). Command errored out with exit status 1: /home/circleci/specs-repo/venv/bin/python3 /home/circleci/specs-repo/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpomv4ghkj Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement py_arkworks_bls12381==0.3.1 (from eth2spec[lint]) (from versions: 0.1.0, 0.1.1, 0.1.2, 0.1.3, 0.1.4, 0.1.5, 0.2.0, 0.2.1, 0.3.0, 0.3.1)
ERROR: No matching distribution found for py_arkworks_bls12381==0.3.1

macOS

CPU: M1 pro
Python: pypy3.9

Building wheels for collected packages: py_arkworks_bls12381
  Building wheel for py_arkworks_bls12381 (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for py_arkworks_bls12381 (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [9 lines of output]
      Running `maturin pep517 build-wheel -i /Users/hww/dev/consensus-specs/venv/bin/python3 --compatibility off`
      🔗 Found pyo3 bindings
      💻 Using `MACOSX_DEPLOYMENT_TARGET=11.0` for aarch64-apple-darwin by default
      error: package `ark-ff v0.4.1` cannot be built because it requires rustc 1.63 or newer, while the currently active rustc version is 1.61.0
      💥 maturin failed
        Caused by: Failed to build a native library through cargo
        Caused by: Cargo build finished with "exit status: 101": `"cargo" "rustc" "--release" "--manifest-path" "/private/var/folders/zf/7y9f1tzn2y33y9601hsthq_40000gn/T/pip-install-m_okx_w7/py-arkworks-bls12381_0cc61e4af6c84c97824e35b118764cfd/Cargo.toml" "--message-format" "json" "--lib" "--" "-C" "link-arg=-undefined" "-C" "link-arg=dynamic_lookup" "-C" "link-args=-Wl,-install_name,@rpath/py_arkworks_bls12381.pypy39-pp73-darwin.so"`
      🐍 Found PyPy 3.9 at /Users/hww/dev/consensus-specs/venv/bin/python3
      Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/Users/hww/dev/consensus-specs/venv/bin/python3', '--compatibility', 'off'] returned non-zero exit status 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for py_arkworks_bls12381
Failed to build py_arkworks_bls12381
ERROR: Could not build wheels for py_arkworks_bls12381, which is required to install pyproject.toml-based projects
WARNING: You are using pip version 22.0.4; however, version 23.0.1 is available.
You should consider upgrading via the '/Users/hww/dev/consensus-specs/venv/bin/python3 -m pip install --upgrade pip' command.
make: *** [install_test] Error 1

IIUC, milagro_bls_binding also uses maturin. It looks like py-arkworks-bls12381 distributions only support CPython 3.11+arm64 now: https://pypi.org/project/py-arkworks-bls12381/#files

@kevaundray could you make py-arkworks-bls12381 support Python >=3.8 (CPython & PyPy) and different platforms by uploading more distributions rather than adding the requirements of installing Rust? 🙏

hwwhww · 2023-02-26T03:50:18Z

@kevaundray do you have some py-ecc v.s. arkworks benchmarks of the pairing? Once we make CircleCI pass, we can try to see if this PR also makes #3255 CircleCI pass.

kevaundray · 2023-02-26T08:51:32Z

@hwwhww I will prepare benchmark numbers + a script to verify and make the modifications that you mentioned above :)

kevaundray · 2023-02-26T12:01:32Z

@hwwhww

Ran benchmarks on my laptop MacBook Pro M1.

Reproduce

To reproduce from source, you can do the following:

Checkout commit
Setup virtual environment
python3 -m examples.benches.bench

To reproduce from pypi:

Download a version >0.3.2

Py-ecc benchmarks

pairingsGen 172.97867500001303 ms
pairings 176.04457500001445 ms
multiplyG1 0.8747412499906204 ms
multiplyG2 3.522136249994219 ms
addG1 0.018061249993479578 ms
addG2 0.07142667000152869 ms
G1_to_bytes48 0.001760420000209706 ms
G2_to_bytes96 0.0599379200139083 ms

Arkworks benchmarks

pairingsGen 1.298829099869181 ms (~133x)
pairings 1.2997874999200576 ms (~135x)
multiplyG1 0.021608329989248887 ms (~40x)
multiplyG2 0.06284999999479624 ms (~58x)
addG1 0.0006941699939488899 ms (~28x)
addG2 0.0021008399926358834 ms (~35x)
G1_to_bytes48 0.00037416999475681223 ms (~4.5x)
G2_to_bytes96 0.0005658299960487057 ms (~104x)

Notes

pairings vs pairingsGen

I store the points in Projective coordinates, but arkworks requires pairing input to be in Affine coordinates. This will incur a cost.

pairingsGen is just the generators being used to which arkworks can optimise as the conversion from Projective to Affine since z=1.

pairingsGen will replace one of the pairings with two points which don't have z=1, to see if there is a noticeable cost.

MultiExp

Having a multi-exponentation algorithm is quite common for ECC libraries so py-arkworks also exposes this. It uses pippenger so would be alot faster than g1_lincomb. We won't be able to use it in the specs but it may be useful for test generation in the future

kevaundray · 2023-02-26T12:05:08Z

I've added a lot more distributions and have updated setup.py here to use 0.3.2

See here for added distributions

hwwhww · 2023-02-26T15:14:31Z

@kevaundray Thank you! CircleCI (Linux) installation succeeds now.

But for macOS, I only see Python 3.10 and Python 3.11 distributions:

py_arkworks_bls12381-0.3.2-cp311-cp311-macosx_11_0_arm64.whl
py_arkworks_bls12381-0.3.2-cp311-cp311-macosx_10_7_x86_64.whl
py_arkworks_bls12381-0.3.2-cp310-cp310-macosx_11_0_arm64.whl
py_arkworks_bls12381-0.3.2-cp310-cp310-macosx_10_7_x86_64.whl

Any chance you could also upload CPython 3.8 and 3.9 support? Python3.8 is the minimum requirement of eth2spec.

I guess it's still early for PyPy+macOS+arm64 distribution so you can skip it for now. (Also, PyPy only provides 3.8 and 3.9 support and started to support M1 last year. 🥲)

I managed to install it with a CPython 3.10 env on the same machine. The test case that used to take me ~30s to run with py-ecc, now is only 6.03s with --bls-type=fastest! 🚀 🚀

kevaundray · 2023-02-26T18:02:15Z

@hwwhww I've pushed a 0.3.3 which now uses 3.8 :)

kevaundray · 2023-02-26T18:10:25Z

@hwwhww if tests are generated using py-arkworks specifically and not via the bls interface, we can probably reduce the six seconds further via py-arkworks specific functions!

hwwhww · 2023-02-27T10:06:37Z

@kevaundray Great!

FYI, just to record my troubleshooting for PyPy, I did something like what I needed for milagro binding:

Upgrade my rustc:
```
rustup update
```

Install py_arkworks_bls12381 via building PyPy wheel:

pip install --no-binary py_arkworks_bls12381 py_arkworks_bls12381

Output:

Collecting py_arkworks_bls12381
Using cached py_arkworks_bls12381-0.3.2.tar.gz (13 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: py_arkworks_bls12381
Building wheel for py_arkworks_bls12381 (pyproject.toml) ... done
Created wheel for py_arkworks_bls12381: filename=py_arkworks_bls12381-0.3.2-pp39-pypy39_pp73-macosx_11_0_arm64.whl size=461544 sha256=89d7fec7125eaf8bd7ba85ce17f02abbca817f1392fc3db7df98eecd6d01b6ec
Stored in directory: /Users/hww/Library/Caches/pip/wheels/4a/c6/f3/3630a2e3deb887d1506a2eaca9de8d6d4e91e8d7d4ddafb559
Successfully built py_arkworks_bls12381
Installing collected packages: py_arkworks_bls12381
Successfully installed py_arkworks_bls12381-0.3.2

The same test case I mentioned (~30s with py-ecc) is now only 3.70s with PyPy3.9 with --bls-type=fastest. 😎

ppopth · 2023-02-27T11:22:51Z

tests/generators/kzg_4844/main.py

@@ -0,0 +1,288 @@
+"""


Should this file be split into many files, like one file per one kzg-4844 function? I think doing it similarly to the one in https://github.com/ethereum/consensus-specs/tree/dev/tests/generators/ssz_generic is more organized.

No strong opinion on this one. @hwwhww what would you prefer? I copied from the bls generators which seem to have this structure

I slightly prefer separate files, but I can see there are some shared helpers/constants. ~500 lines look acceptable so far, but if you expect to extend the test cases, refactoring them early seems better.

Decided on the call today to leave for now and split when the file becomes too large.

Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com>

jtraglia

There are two minor typos that need to be fixed here:

tests/generators/kzg_4844/main.py

tests/formats/kzg/blob_to_kzg_commitment.md

tests/formats/kzg/compute_blob_kzg_proof.md

tests/formats/kzg/compute_kzg_proof.md

tests/formats/kzg/verify_blob_kzg_proof.md

tests/formats/kzg/verify_blob_kzg_proof_batch.md

tests/formats/kzg/verify_kzg_proof.md

hwwhww · 2023-03-01T05:21:27Z

tests/formats/kzg/verify_blob_kzg_proof.md

+
+## Condition
+
+The `verify_blob_kzg_proof` handler should verify that `commitment` is a correct KZG commitment to `blob` by using the blob KZG proof `proof`, and the result should match the expected `output`. If the commitment or proof are invalid (e.g. not on the curve or not in the G1 subgroup of the BLS curve) or `blob` is invalid (e.g. incorrect length or one of the 32 byte blocks does not represent a BLS field element), it should error, i.e. the output should be `None`.


If the commitment or proof is invalid (e.g. not on the curve or not in the G1 subgroup of the BLS curve) or blob is invalid (e.g. incorrect length or one of the 32-byte blocks does not represent a BLS field element), it should error, i.e. the output should be null.

Note that it's not as same as the pyspec BLS signature verify_* methods: in BLS signature API implementation, we catch the exception and return False.

I'm not suggesting adding error handling in polynomial-commitments.md, I think that's too Pythonic. Just note that the clients have to deal with the error handling themselves.

btw, does c-kzg also throw exceptions in these cases? /cc @jtraglia

I'm not entirely sure what cases you're referring to, but the C library in c-kzg-4844 will return C_KZG_BADARGS if given invalid inputs. Generally, the bindings will throw an exception if the library doesn't return C_KZG_OK (success).

For example, the case that the input is in the correct length but doesn't pass the subgroup check. I suppose it's not C_KZG_OK there so will throw an exception?

Yes, the bindings will throw an exception. For example:

https://github.com/ethereum/c-kzg-4844/blob/ae0aeaf33d4433bc4c25e10bffc59cf7f1aa6650/bindings/java/c_kzg_4844_jni.c#L207-L211

jtraglia · 2023-03-01T21:27:30Z

So either there's something wrong with the tests or something wrong with ckzg. When tested against ckzg, there are failures in compute_blob_kzg_proof, verify_blob_kzg_proof, and verify_blob_kzg_proof_batch, such as:

AssertionError: ../../newtests/compute_blob_kzg_proof/small/compute_blob_kzg_proof_case_valid_blob_94b2a33de83f0ee5/data.yaml
proof.hex()='97425e513a683f1d90a8ebadf84b8f60576e41270272a5561714c450fc904e9eb44f15d8ea37f7672739e82a8f5eb671'
expected_proof.hex()='b1f94037a955facd0540d16dc23533d7f6e6fbe0573af591ca6a2a641edbb8fe3f7feb1df3d82d739f08d99a4abca36c'

AssertionError: ../../newtests/verify_blob_kzg_proof/small/verify_blob_kzg_proof_case_correct_proof_5caa8bb4962217cb/data.yaml
valid=False
expected_valid=True

AssertionError: ../../newtests/verify_blob_kzg_proof_batch/small/verify_blob_kzg_proof_batch_case_9176f3148be71345/data.yaml
valid=False
expected_valid=True

All of the test cases for the other functions pass.

Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com>

hwwhww

I think it's fine to merge it now and then @kevaundray can add more bls.py fixes later. Please use squash and merge for the 45 commits. 🙏

But also see @jtraglia pointed out that the new test vector found inconsistency between pyspec or c-kzg, should we address that in this PR? (or the issue may be in c-kzg?)

asn-d6 · 2023-03-02T13:26:24Z

So either there's something wrong with the tests or something wrong with ckzg. When tested against ckzg, there are failures in compute_blob_kzg_proof, verify_blob_kzg_proof, and verify_blob_kzg_proof_batch, such as:

AssertionError: ../../newtests/compute_blob_kzg_proof/small/compute_blob_kzg_proof_case_valid_blob_94b2a33de83f0ee5/data.yaml
proof.hex()='97425e513a683f1d90a8ebadf84b8f60576e41270272a5561714c450fc904e9eb44f15d8ea37f7672739e82a8f5eb671'
expected_proof.hex()='b1f94037a955facd0540d16dc23533d7f6e6fbe0573af591ca6a2a641edbb8fe3f7feb1df3d82d739f08d99a4abca36c'

AssertionError: ../../newtests/verify_blob_kzg_proof/small/verify_blob_kzg_proof_case_correct_proof_5caa8bb4962217cb/data.yaml
valid=False
expected_valid=True

AssertionError: ../../newtests/verify_blob_kzg_proof_batch/small/verify_blob_kzg_proof_batch_case_9176f3148be71345/data.yaml
valid=False
expected_valid=True

All of the test cases for the other functions pass.

Seems like a bug was found -- ckzg was not following the spec! See ethereum/c-kzg-4844#168 for a fix!

With that PR I'm not getting the errors above anymore (using the python bindings).

asn-d6 · 2023-03-02T13:26:49Z

That was a c

I think it's fine to merge it now and then @kevaundray can add more bls.py fixes later. Please use squash and merge for the 45 commits. pray

But also see @jtraglia pointed out that the new test vector found inconsistency between pyspec or c-kzg, should we address that in this PR? (or the issue may be in c-kzg?)

that was a c-kzg bug after all!

tests/generators/kzg_4844/main.py

kevaundray and others added 10 commits February 24, 2023 13:15

bls.py : Abstract away usage of py_ecc methods

b2b7a98

polynomial-commitments: Remove usage of concrete py_ecc types

773addb

fix lint

b2d413e

Arkworks integration

636c572

add py_arkworks to setup.py

4a61089

add setuptools-rust to see if it adds the rust compiler

f59dd7c

remove setuptools: seems easier to install rust via the CI

d8e0cf1

Fix bug when using milagro bls

1a34476

Add new 'fastest' BLS option that combines arkworks and milagro

dbfb1ac

Test generators for compute_kzg_proof

df9167f

update to use 0.3.2

55e00f3

kevaundray and others added 2 commits February 26, 2023 12:10

fix linter

882fc72

Bump py_arkworks_bls12381 to v0.3.2

927f673

kevaundray and others added 5 commits February 26, 2023 18:27

upgrade to 0.3.3

ca27126

More test generators for 4844

e4c938d

Merge branch 'kw/py-ecc-agnostic-4844' into kzg-4844-generators

e47f0a8

Lint

5d8d9c8

Whitespace

70de749

ppopth reviewed Feb 27, 2023

View reviewed changes

dankrad added 2 commits February 27, 2023 16:57

Add tests for verify_blob_kzg_proof

8cb5976

Add tests for compute_blob_kzg_proof

ecd7654

dankrad and others added 3 commits February 28, 2023 12:27

Update tests/generators/kzg_4844/main.py

ce6a3fd

Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com>

Long line

fd4c189

minor nits from code review

7c71baf

jtraglia requested changes Feb 28, 2023

View reviewed changes

tests/generators/kzg_4844/main.py Outdated Show resolved Hide resolved

tests/generators/kzg_4844/main.py Outdated Show resolved Hide resolved

dankrad and others added 4 commits February 28, 2023 20:37

Plurals

66c1e14

Merge branch 'dev' into kzg-4844-generators

e5dcb36

Merge branch 'kw/py-ecc-agnostic-4844' into kzg-4844-generators

cad12c1

Add test format docs

f41cf35

hwwhww reviewed Mar 1, 2023

View reviewed changes

dankrad and others added 6 commits March 1, 2023 22:09

Update tests/formats/kzg/blob_to_kzg_commitment.md

0768d52

Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com>

Update tests/formats/kzg/compute_blob_kzg_proof.md

def5c48

Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com>

Update tests/formats/kzg/verify_blob_kzg_proof.md

aaf00c7

Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com>

Update tests/formats/kzg/compute_kzg_proof.md

69e7d18

Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com>

Update tests/formats/kzg/verify_blob_kzg_proof_batch.md

c23fe5f

Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com>

Update tests/formats/kzg/verify_kzg_proof.md

695b49a

Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com>

hwwhww approved these changes Mar 2, 2023

View reviewed changes

ppopth reviewed Mar 2, 2023

View reviewed changes

tests/generators/kzg_4844/main.py Outdated Show resolved Hide resolved

ppopth reviewed Mar 2, 2023

View reviewed changes

tests/generators/kzg_4844/main.py Outdated Show resolved Hide resolved

dankrad added 2 commits March 2, 2023 15:27

INVALID_BLOB -> blob

f1c3c7d

another INVALID_BLOB -> blob

d52da28

ppopth reviewed Mar 2, 2023

View reviewed changes

tests/generators/kzg_4844/main.py Outdated Show resolved Hide resolved

dankrad added 2 commits March 2, 2023 20:19

Simplify incorrect proof in batch test

01873c0

Lint

c497ef9

dankrad merged commit 86fb82b into dev Mar 2, 2023

dankrad deleted the kzg-4844-generators branch March 2, 2023 20:49

kevaundray mentioned this pull request Mar 3, 2023

EIP4844: Remove reliance on py-ecc types in polynomial-commitments and add py-arkworks #3272

Closed

belijzajac mentioned this pull request Mar 9, 2023

Update to c-kzg-4844 upstream grandinetech/rust-kzg#191

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test generators for kzg-4844 libraries #3274

Test generators for kzg-4844 libraries #3274

dankrad commented Feb 25, 2023 •

edited

Loading

hwwhww commented Feb 26, 2023 •

edited

Loading

hwwhww commented Feb 26, 2023

kevaundray commented Feb 26, 2023

kevaundray commented Feb 26, 2023 •

edited

Loading

kevaundray commented Feb 26, 2023

hwwhww commented Feb 26, 2023 •

edited

Loading

kevaundray commented Feb 26, 2023

kevaundray commented Feb 26, 2023

hwwhww commented Feb 27, 2023

ppopth Feb 27, 2023

dankrad Feb 27, 2023

hwwhww Feb 28, 2023

dankrad Feb 28, 2023

jtraglia left a comment

hwwhww Mar 1, 2023

jtraglia Mar 1, 2023

hwwhww Mar 2, 2023

jtraglia Mar 2, 2023

jtraglia commented Mar 1, 2023 •

edited

Loading

hwwhww left a comment •

edited

Loading

asn-d6 commented Mar 2, 2023 •

edited

Loading

asn-d6 commented Mar 2, 2023


		## Condition

		The `verify_blob_kzg_proof` handler should verify that `commitment` is a correct KZG commitment to `blob` by using the blob KZG proof `proof`, and the result should match the expected `output`. If the commitment or proof are invalid (e.g. not on the curve or not in the G1 subgroup of the BLS curve) or `blob` is invalid (e.g. incorrect length or one of the 32 byte blocks does not represent a BLS field element), it should error, i.e. the output should be `None`.

Test generators for kzg-4844 libraries #3274

Test generators for kzg-4844 libraries #3274

Conversation

dankrad commented Feb 25, 2023 • edited Loading

hwwhww commented Feb 26, 2023 • edited Loading

Linux (CircleCI)

macOS

hwwhww commented Feb 26, 2023

kevaundray commented Feb 26, 2023

kevaundray commented Feb 26, 2023 • edited Loading

Reproduce

Py-ecc benchmarks

Arkworks benchmarks

Notes

kevaundray commented Feb 26, 2023

hwwhww commented Feb 26, 2023 • edited Loading

kevaundray commented Feb 26, 2023

kevaundray commented Feb 26, 2023

hwwhww commented Feb 27, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jtraglia left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jtraglia commented Mar 1, 2023 • edited Loading

hwwhww left a comment • edited Loading

Choose a reason for hiding this comment

asn-d6 commented Mar 2, 2023 • edited Loading

asn-d6 commented Mar 2, 2023

dankrad commented Feb 25, 2023 •

edited

Loading

hwwhww commented Feb 26, 2023 •

edited

Loading

kevaundray commented Feb 26, 2023 •

edited

Loading

hwwhww commented Feb 26, 2023 •

edited

Loading

jtraglia commented Mar 1, 2023 •

edited

Loading

hwwhww left a comment •

edited

Loading

asn-d6 commented Mar 2, 2023 •

edited

Loading