feat: add vector storage benchmarks for EVM storage operations #1734
Conversation
Implements parametrized storage benchmarks to measure gas costs for different storage state transitions: - Cold writes (0 -> non-zero): ~20,000 gas per slot - Warm updates (non-zero -> non-zero): ~2,900 gas per slot - Storage clearing (non-zero -> 0): ~2,900 gas + refund The implementation uses a single contract that accepts parameters via calldata, pre-deployed with 500 filled slots to enable testing all transition types. This provides accurate gas measurements with minimal loop overhead (~39 gas/slot). Tests are marked as stateful to avoid gas exhaustion expectations and work with the execution-specs testing framework.
| 0, # Start at slot 0 | ||
| 0xBEEFBEEF, # Different value | ||
| id="nonzero_to_nonzero" | ||
| ), |
There was a problem hiding this comment.
Could you add the situation zero to zero here also? This might sound weird, but because some clients are slow on reading non-existent storage slots, this might be a bad situation for those.
| pytest.param( | ||
| "nonzero_to_nonzero", | ||
| 0, # Start at slot 0 | ||
| 0xBEEFBEEF, # Different value |
There was a problem hiding this comment.
For maximal load we want to use the longest values here (0xffff..ff) and to ensure clients have to hash everything, it would also help if we would change the value for each slot (so start at 0xff..ff and subtract one each time) (Can be a different PR since this needs changes to the contract code)
There was a problem hiding this comment.
Notice that changing the value will imply extra costs. Though we can pass the different values via calldata maybe. Let me see what to do here
| bytecode += Op.PUSH1(0) + Op.CALLDATALOAD # num_slots at bytes 0-31 | ||
| bytecode += Op.PUSH1(32) + Op.CALLDATALOAD # start_slot at bytes 32-63 | ||
| bytecode += Op.PUSH1(64) + Op.CALLDATALOAD # value at bytes 64-95 |
There was a problem hiding this comment.
small suggestion:
| bytecode += Op.PUSH1(0) + Op.CALLDATALOAD # num_slots at bytes 0-31 | |
| bytecode += Op.PUSH1(32) + Op.CALLDATALOAD # start_slot at bytes 32-63 | |
| bytecode += Op.PUSH1(64) + Op.CALLDATALOAD # value at bytes 64-95 | |
| bytecode += Op.CALLDATALOAD(0) # num_slots at bytes 0-31 | |
| bytecode += Op.CALLDATALOAD(32) # start_slot at bytes 32-63 | |
| bytecode += Op.CALLDATALOAD(64) # value at bytes 64-95 |
| 2. nonzero to 0: Clearing existing slots (provides gas refund) | ||
| 3. nonzero to nonzero: Updating warm slots (moderate cost) | ||
| """ | ||
| sender = pre.fund_eoa(10**18) |
There was a problem hiding this comment.
| sender = pre.fund_eoa(10**18) | |
| sender = pre.fund_eoa() |
| # Create transaction to call the contract | ||
| # Use a reasonable gas limit that covers the operation | ||
| # Each SSTORE costs up to 20,000 gas for cold writes plus overhead | ||
| gas_limit = 21000 + (num_slots * 50000) # Base tx cost + storage operations |
There was a problem hiding this comment.
I am curious why specifying the gas limit here, will it be affected if we make it the default value?
If necessary, i would suggest to use:
| gas_limit = 21000 + (num_slots * 50000) # Base tx cost + storage operations | |
| intrinsic_gas_cost_calc = fork.transaction_intrinsic_cost_calculator() | |
| gas_limit = intrinsic_gas_cost_calc() + (num_slots * 50000) # Base tx cost + storage operations |
| def test_storage_transitions_benchmark( | ||
| blockchain_test: BlockchainTestFiller, | ||
| pre: Alloc, | ||
| fork: Fork, | ||
| num_slots: int, | ||
| transition_type: str, | ||
| start_offset: int, | ||
| write_value: int, | ||
| ) -> None: |
There was a problem hiding this comment.
Unused variable
| def test_storage_transitions_benchmark( | |
| blockchain_test: BlockchainTestFiller, | |
| pre: Alloc, | |
| fork: Fork, | |
| num_slots: int, | |
| transition_type: str, | |
| start_offset: int, | |
| write_value: int, | |
| ) -> None: | |
| def test_storage_transitions_benchmark( | |
| blockchain_test: BlockchainTestFiller, | |
| pre: Alloc, | |
| num_slots: int, | |
| start_offset: int, | |
| write_value: int, | |
| ) -> None: |
|
|
||
|
|
||
| @pytest.mark.valid_from("Prague") | ||
| @pytest.mark.stateful # Mark as stateful instead of benchmark |
There was a problem hiding this comment.
| @pytest.mark.stateful # Mark as stateful instead of benchmark |
All the tests under statful/ folder would be marked as stateful test! No need to specify them.
| @pytest.mark.stateful # Mark as stateful instead of benchmark | ||
| @pytest.mark.parametrize("num_slots", [1, 10, 50, 100, 200]) | ||
| @pytest.mark.parametrize( | ||
| "transition_type,start_offset,write_value", |
There was a problem hiding this comment.
I wonder if we still need transition_type parametrization? It seems similar to the id, and this value is not referenced in the test implementation.
|
Note: please run |
3 participants
Implements parametrized storage benchmarks to measure gas costs for different storage state transitions:
The implementation uses a single contract that accepts parameters via calldata, pre-deployed with 500 filled slots to enable testing all transition types. This provides accurate gas measurements with minimal loop overhead (~39 gas/slot).
Tests are marked as stateful to avoid gas exhaustion expectations and work with the execution-specs testing framework.
✅ Checklist
toxchecks to avoid unnecessary CI fails, see also Code Standards and Enabling Pre-commit Checks:uvx tox -e statictype(scope):.mkdocs servelocally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.@ported_frommarker.Cute Animal Picture