Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimized validate_blob_transaction_wrapper() function #4

Closed
wants to merge 5 commits into from

Conversation

asn-d6
Copy link

@asn-d6 asn-d6 commented Mar 1, 2022

// Verify that the list of `blobs` map to the list of `commitments`
//
// This is an optimization over the naive approach (written in the EIP) of iteratively checking each blob against each
// commitment.  The naive approach requires n*l scalar multiplications where `n` is the number of blobs and `l` is
// FIELD_ELEMENTS_PER_BLOB to compute the commitments for all blobs.
//
// A more efficient approach is to build a linear combination of all blobs and commitments and check all of them in a
// single multi-scalar multiplication.
//
// The MSM would look like this (for three blobs with two field elements each):
//     r_0(b0_0*L_0 + b0_1*L_1) + r_1(b1_0*L_0 + b1_1*L_1) + r_2(b2_0*L_0 + b2_1*L_1)
// which we would need to check against the linear combination of commitments: r_0*C_0 + r_1*C_1 + r_2*C_2
// In the above, `r` are the random scalars of the linear combination, `b0` is the zero blob, `L` are the elements
// of the KZG_SETUP_LAGRANGE and `C` are the commitments provided.
//
// By re-grouping the above equation around the `L` points we can reduce the amount of scalar multiplications further
// (down to just `n` scalar multiplications) by making the MSM look like this:
//     (r_0*b0_0 + r_1*b1_0 + r_2*b2_0) * L_0 + (r_0*b0_1 + r_1*b1_1 + r_2*b2_1) * L_1
func VerifyBlobs(commitments []*bls.G1Point, blobs [][]bls.Fr) error

Same logic should be used for verify_blobs() on the CL.

@asn-d6 asn-d6 mentioned this pull request Mar 1, 2022
@asn-d6 asn-d6 changed the title Optimized verify_blobs() function Optimized validate_blob_transaction_wrapper() function Mar 1, 2022
@protolambda
Copy link
Owner

Awesome, Merged into #1 (github doesn't seem to detect the rebase/merge though):

  • Rebased onto datablobs
  • Cleaned up the Blobs() and Points() methods (Blob is an array type now, and made Blob.Parse() method, and renamed the other point/fr parse methods to Parse()), and reduce allocations
  • Updated test to reflect the slice->array type change of Blob, had to fix a mutation on old copy

asn-d6 added a commit to asn-d6/EIPs that referenced this pull request May 11, 2022
To validate a 4844 transaction in the mempool, the verifier checks that each provided KZG commitment matches the
polynomial represented by the corresponding blob data.

     | d_1 | d_2 | d_3 | ... | d_4096 |    -> commitment

Before this patch, to do this validation, we reconstructed the commitment from the blob data (d_i above), and checked
it against the provided commitment. This was expensive because computing a commitment from blob data (even using
Lagrange basis) involves N scalar multiplications, where N is the number of field elements per blob.

Initial benchmarking showed that this was about 40ms for N=4096 which was deemed too expensive. For more details see:
             https://hackmd.io/@protolambda/eip-4844-implementer-notes#Optimizations
             protolambda/go-ethereum#4

In this patch, we speed this up by providing a KZG proof for each commitment. The verifier can check that proof to
ensure that the KZG commitment matches the polynomial represented by the corresponding blob data.

     | d_1 | d_2 | d_3 | ... | d_4096 |    -> commitment, proof

To do so, we evaluate the blob data polynomial at a random point `x` to get a value `y`. We then use the KZG proof to
ensure that the commited polynomial (i.e. the commitment) also evaluates to `y` at `x`. If the check passes, it means
that the KZG commitment matches the polynomial represented by the blob data.

This is significantly faster since evaluating the blob data polynomial at a random point using the Barycentric formula
can be done efficiently with only field operations (see https://hackmd.io/@vbuterin/barycentric_evaluation). Then,
verifying a KZG proof takes two pairing operations (which take about 0.6ms each). This brings the total verification
cost to about 2 ms per blob.

With some additional optimizations (using linear combination tricks as the ones linked above) we can batch all the
blobs together into a single efficient verification, and hence verify the entire transaction in 2.5 ms. The same
techniques can be used to efficiently verify blocks on the consensus side.
asn-d6 added a commit to asn-d6/EIPs that referenced this pull request May 11, 2022
To validate a 4844 transaction in the mempool, the verifier checks that each provided KZG commitment matches the
polynomial represented by the corresponding blob data.

     | d_1 | d_2 | d_3 | ... | d_4096 |    -> commitment

Before this patch, to do this validation, we reconstructed the commitment from the blob data (d_i above), and checked
it against the provided commitment. This was expensive because computing a commitment from blob data (even using
Lagrange basis) involves N scalar multiplications, where N is the number of field elements per blob.

Initial benchmarking showed that this was about 40ms for N=4096 which was deemed too expensive. For more details see:
             https://hackmd.io/@protolambda/eip-4844-implementer-notes#Optimizations
             protolambda/go-ethereum#4

In this patch, we speed this up by providing a KZG proof for each commitment. The verifier can check that proof to
ensure that the KZG commitment matches the polynomial represented by the corresponding blob data.

     | d_1 | d_2 | d_3 | ... | d_4096 |    -> commitment, proof

To do so, we evaluate the blob data polynomial at a random point `x` to get a value `y`. We then use the KZG proof to
ensure that the commited polynomial (i.e. the commitment) also evaluates to `y` at `x`. If the check passes, it means
that the KZG commitment matches the polynomial represented by the blob data.

This is significantly faster since evaluating the blob data polynomial at a random point using the Barycentric formula
can be done efficiently with only field operations (see https://hackmd.io/@vbuterin/barycentric_evaluation). Then,
verifying a KZG proof takes two pairing operations (which take about 0.6ms each). This brings the total verification
cost to about 2 ms per blob.

With some additional optimizations (using linear combination tricks as the ones linked above) we can batch all the
blobs together into a single efficient verification, and hence verify the entire transaction in 2.5 ms. The same
techniques can be used to efficiently verify blocks on the consensus side.
asn-d6 added a commit to asn-d6/EIPs that referenced this pull request May 23, 2022
To validate a 4844 transaction in the mempool, the verifier checks that each provided KZG commitment matches the
polynomial represented by the corresponding blob data.

     | d_1 | d_2 | d_3 | ... | d_4096 |    -> commitment

Before this patch, to do this validation, we reconstructed the commitment from the blob data (d_i above), and checked
it against the provided commitment. This was expensive because computing a commitment from blob data (even using
Lagrange basis) involves N scalar multiplications, where N is the number of field elements per blob.

Initial benchmarking showed that this was about 40ms for N=4096 which was deemed too expensive. For more details see:
             https://hackmd.io/@protolambda/eip-4844-implementer-notes#Optimizations
             protolambda/go-ethereum#4

In this patch, we speed this up by providing a KZG proof for each commitment. The verifier can check that proof to
ensure that the KZG commitment matches the polynomial represented by the corresponding blob data.

     | d_1 | d_2 | d_3 | ... | d_4096 |    -> commitment, proof

To do so, we evaluate the blob data polynomial at a random point `x` to get a value `y`. We then use the KZG proof to
ensure that the commited polynomial (i.e. the commitment) also evaluates to `y` at `x`. If the check passes, it means
that the KZG commitment matches the polynomial represented by the blob data.

This is significantly faster since evaluating the blob data polynomial at a random point using the Barycentric formula
can be done efficiently with only field operations (see https://hackmd.io/@vbuterin/barycentric_evaluation). Then,
verifying a KZG proof takes two pairing operations (which take about 0.6ms each). This brings the total verification
cost to about 2 ms per blob.

With some additional optimizations (using linear combination tricks as the ones linked above) we can batch all the
blobs together into a single efficient verification, and hence verify the entire transaction in 2.5 ms. The same
techniques can be used to efficiently verify blocks on the consensus side.
asn-d6 added a commit to asn-d6/EIPs that referenced this pull request Jun 29, 2022
To validate a 4844 transaction in the mempool, the verifier checks that each provided KZG commitment matches the
polynomial represented by the corresponding blob data.

     | d_1 | d_2 | d_3 | ... | d_4096 |    -> commitment

Before this patch, to do this validation, we reconstructed the commitment from the blob data (d_i above), and checked
it against the provided commitment. This was expensive because computing a commitment from blob data (even using
Lagrange basis) involves N scalar multiplications, where N is the number of field elements per blob.

Initial benchmarking showed that this was about 40ms for N=4096 which was deemed too expensive. For more details see:
             https://hackmd.io/@protolambda/eip-4844-implementer-notes#Optimizations
             protolambda/go-ethereum#4

In this patch, we speed this up by providing a KZG proof for each commitment. The verifier can check that proof to
ensure that the KZG commitment matches the polynomial represented by the corresponding blob data.

     | d_1 | d_2 | d_3 | ... | d_4096 |    -> commitment, proof

To do so, we evaluate the blob data polynomial at a random point `x` to get a value `y`. We then use the KZG proof to
ensure that the commited polynomial (i.e. the commitment) also evaluates to `y` at `x`. If the check passes, it means
that the KZG commitment matches the polynomial represented by the blob data.

This is significantly faster since evaluating the blob data polynomial at a random point using the Barycentric formula
can be done efficiently with only field operations (see https://hackmd.io/@vbuterin/barycentric_evaluation). Then,
verifying a KZG proof takes two pairing operations (which take about 0.6ms each). This brings the total verification
cost to about 2 ms per blob.

With some additional optimizations (using linear combination tricks as the ones linked above) we can batch all the
blobs together into a single efficient verification, and hence verify the entire transaction in 2.5 ms. The same
techniques can be used to efficiently verify blocks on the consensus side.
kodiakhq bot pushed a commit to ethereum/EIPs that referenced this pull request Jun 29, 2022
…ofs) (#5088)

* Fix missing variables/funcs in validate_blob_transaction_wrapper()

There is no `tx.message.blob_commitments` anymore, or `kzg_to_commitment()`

* Introduce KZGProof as its own type instead of using KZGCommitment

* Introduce high-level logic of new efficient transaction validation

To validate a 4844 transaction in the mempool, the verifier checks that each provided KZG commitment matches the
polynomial represented by the corresponding blob data.

     | d_1 | d_2 | d_3 | ... | d_4096 |    -> commitment

Before this patch, to do this validation, we reconstructed the commitment from the blob data (d_i above), and checked
it against the provided commitment. This was expensive because computing a commitment from blob data (even using
Lagrange basis) involves N scalar multiplications, where N is the number of field elements per blob.

Initial benchmarking showed that this was about 40ms for N=4096 which was deemed too expensive. For more details see:
             https://hackmd.io/@protolambda/eip-4844-implementer-notes#Optimizations
             protolambda/go-ethereum#4

In this patch, we speed this up by providing a KZG proof for each commitment. The verifier can check that proof to
ensure that the KZG commitment matches the polynomial represented by the corresponding blob data.

     | d_1 | d_2 | d_3 | ... | d_4096 |    -> commitment, proof

To do so, we evaluate the blob data polynomial at a random point `x` to get a value `y`. We then use the KZG proof to
ensure that the commited polynomial (i.e. the commitment) also evaluates to `y` at `x`. If the check passes, it means
that the KZG commitment matches the polynomial represented by the blob data.

This is significantly faster since evaluating the blob data polynomial at a random point using the Barycentric formula
can be done efficiently with only field operations (see https://hackmd.io/@vbuterin/barycentric_evaluation). Then,
verifying a KZG proof takes two pairing operations (which take about 0.6ms each). This brings the total verification
cost to about 2 ms per blob.

With some additional optimizations (using linear combination tricks as the ones linked above) we can batch all the
blobs together into a single efficient verification, and hence verify the entire transaction in 2.5 ms. The same
techniques can be used to efficiently verify blocks on the consensus side.

* Introduce polynomial helper functions for transaction validation

* Implement high-level logic of aggregated proof verification

* Add helper functions for aggregated proof verification

Also abstract `lincomb()` out of the `blob_to_kzg()` function to be used in the verification.

* Fixes after review on the consensus PR
protolambda pushed a commit that referenced this pull request Sep 16, 2022
Add BlobVersionedHashes to JSON-RPC Transaction result
protolambda pushed a commit that referenced this pull request Sep 16, 2022
The mint field was not being reported on deposit transactions over
the JSON RPC interface because the field was misnamed.
protolambda added a commit that referenced this pull request Sep 16, 2022
protolambda pushed a commit that referenced this pull request Jan 4, 2023
The mint field was not being reported on deposit transactions over
the JSON RPC interface because the field was misnamed.
protolambda added a commit that referenced this pull request Jan 4, 2023
nachomazzara pushed a commit to nachomazzara/EIPs that referenced this pull request Jan 13, 2023
…ofs) (ethereum#5088)

* Fix missing variables/funcs in validate_blob_transaction_wrapper()

There is no `tx.message.blob_commitments` anymore, or `kzg_to_commitment()`

* Introduce KZGProof as its own type instead of using KZGCommitment

* Introduce high-level logic of new efficient transaction validation

To validate a 4844 transaction in the mempool, the verifier checks that each provided KZG commitment matches the
polynomial represented by the corresponding blob data.

     | d_1 | d_2 | d_3 | ... | d_4096 |    -> commitment

Before this patch, to do this validation, we reconstructed the commitment from the blob data (d_i above), and checked
it against the provided commitment. This was expensive because computing a commitment from blob data (even using
Lagrange basis) involves N scalar multiplications, where N is the number of field elements per blob.

Initial benchmarking showed that this was about 40ms for N=4096 which was deemed too expensive. For more details see:
             https://hackmd.io/@protolambda/eip-4844-implementer-notes#Optimizations
             protolambda/go-ethereum#4

In this patch, we speed this up by providing a KZG proof for each commitment. The verifier can check that proof to
ensure that the KZG commitment matches the polynomial represented by the corresponding blob data.

     | d_1 | d_2 | d_3 | ... | d_4096 |    -> commitment, proof

To do so, we evaluate the blob data polynomial at a random point `x` to get a value `y`. We then use the KZG proof to
ensure that the commited polynomial (i.e. the commitment) also evaluates to `y` at `x`. If the check passes, it means
that the KZG commitment matches the polynomial represented by the blob data.

This is significantly faster since evaluating the blob data polynomial at a random point using the Barycentric formula
can be done efficiently with only field operations (see https://hackmd.io/@vbuterin/barycentric_evaluation). Then,
verifying a KZG proof takes two pairing operations (which take about 0.6ms each). This brings the total verification
cost to about 2 ms per blob.

With some additional optimizations (using linear combination tricks as the ones linked above) we can batch all the
blobs together into a single efficient verification, and hence verify the entire transaction in 2.5 ms. The same
techniques can be used to efficiently verify blocks on the consensus side.

* Introduce polynomial helper functions for transaction validation

* Implement high-level logic of aggregated proof verification

* Add helper functions for aggregated proof verification

Also abstract `lincomb()` out of the `blob_to_kzg()` function to be used in the verification.

* Fixes after review on the consensus PR
protolambda added a commit that referenced this pull request Mar 3, 2023
This commit squashes the op-geth fork history into a more maintainable
diff for rebasing upon upstream geth.

reference-optimistic-geth changes (origins of op-geth in early Bedrock
development stage):
- Deposit TX Type
- Enable deposit tx in EVM/tx pool
- Change deposit nonce to not be the max nonce
- Extend PayloadAttributesV1 with a Transactions field
- Force deposits at the start of each L2 block
- Fix height check
- noTxPool flag, reproduce block in verifier mode without tx pool interference
- Fix RPC json marshalling (ref op-geth PR 4)
- Deposit txs block height check in block body validation (ref op-geth PR 5)
- core: do not try to reinject deposit txs into tx-pool (ref-op-geth PR 6)
- deposit source hash field instead of L2 block height and tx index combination
- Include invalid deposits, rewind state, but always persist mint (ethereum#10)
- Provide gas to Call/Create in deposit transactions (ethereum#12)
- Add docker builds (ref-op-geth PR 16, 17)
- Don't panic on deposit transaction signature values or chain ID (ref-op-geth PR 18)
- core: Add version to DepositTx (ref-op-geth PR 19)
- Enable Geth build/lint/test in CircleCI (ref-op-geth PR 23)
- core: Include guaranteed gas in the gas pool (ref-op-geth PR 21)
- core: handle base fee, l1 availability fee, tx fee (ref-op-geth PR 27)
- fix: deposit tx hash
- fix l1 fee cache, rpc, tracing and tx pool
- core: remove deposit-tx sub-type (a.k.a. deposit version byte)
- eth/catalyst: allow engine user to reorg own chain
- miner: restore ability to reorg deep as block builder
- params: print Optimism consensus type in banner
- core/types: remove unused protected() method, see upstream PR 23376
- core: do not mutate original balance value in tx pool l1 cost adjustment
- core: subtract deposit gas from pool, so other txs do not use the same gas. And fail tx processing if deposits reach gas limit
- core/types: deposits do not tip, avoid basefee subtraction
- Unmeter the L1 Attributes Transaction
- miner: handle force tx errors as critical, clean up diff
- ci: Switch branch
- eth,miner: return STATUS_INVALID when failing to process forced transactions in request (ref-op-geth PR 40)
- verifier: forward tx to sequencer based on flag
- txpool: add flag to disable tx gossip (ref-op-geth PR 42)
- Add op-geth version in addition to geth version (ref-op-geth PR 43)
- ci: CircleCI improvements (ref-op-geth PR 44)
- Rename to op-geth
- Build latest tag on optimism branch

op-geth changes:
- Expose cache config in simulated backend (#2)
- Add EIP-1559 parameters
- eth/catalyst: update payload id computation (#1)
- make eip1559 configurable (#4)
- post-merge network should not log warnings about missing transition information (#5)
- Make the simulator more configurable (ethereum#6)
- fix OPB-6 - IsDepositTx check instead of artificial nonce value check (ethereum#7)
- Simulated backend - enable proof of stake consensus type and fix performance issue (ethereum#8)
- accounts: simulated backend consensus engine option and immediate tx indexing
- consensus/beacon: recognize all blocks as reached TTD with 0 TTD in chain config
- Add --rollup.historicalhttp CLI flag and fix backend iface
- Flags and interfaces for historical RPC requests (ethereum#12)
- Redirect historical RPC requests (ethereum#13)
- Use the pre-existing ethereum.NotFound error (ethereum#18)
- Add historical endpoint to TraceBlockByNumber and TraceBlockByHash (ethereum#19)
- Add historical endpoint to TraceTransaction (ethereum#20)
- Add historical endpoint to TraceCall (ethereum#21)
- optimism: fee params from info txi, update l1 cost func GPO params read (ethereum#15)
- add hardcoded addresses for fee payouts (ethereum#23)
- dynamic gas limit via engine API (ethereum#22)

Co-authored-by: Matthew Slipper <me@matthewslipper.com>
Co-authored-by: Joshua Gutow <jgutow@oplabs.co>
Co-authored-by: protolambda <proto@protolambda.com>
Co-authored-by: Mark Tyneway <mark.tyneway@gmail.com>
Co-authored-by: Maurelian <maurelian@protonmail.ch>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants