Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hive: Add --sim.exactmatch flag #969

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

spencer-tb
Copy link
Contributor

Description

Following this PR: #929

On occasion we may use a regex pattern that will run more tests than intended.

The intention behind the sim.limit below may be to solely run a single test in isolation:

... --sim.limit engine-cancun/"InvalidPayloadAttributes: Missing BeaconRoot" --docker.output

Due to the increasing number of tests within the engine simulator alongside its programtic naming of tests, the above sim.limit will run and additional test using regex matching:

InvalidPayloadAttributes: Missing BeaconRoot
InvalidPayloadAttributes: Missing BeaconRoot (Syncing)

Updated Proposed Change

Adds an additional boolean flag: --sim.exactmatch, that can be used to escape and anchor the regex expression specified within --sim.limit.

@spencer-tb spencer-tb force-pushed the hive/add-sim-exactmatch-flag branch from 8632ef1 to b3a2984 Compare March 17, 2025 12:10
@spencer-tb
Copy link
Contributor Author

@danceratopz @marioevz - is this valuable? Maybe useful for non-consume based simulators. Otherwise I can update the docs to show how you can use regex correctly for exact matches within --sim.limit.

@danceratopz
Copy link
Member

@danceratopz @marioevz - is this valuable? Maybe useful for non-consume based simulators. Otherwise I can update the docs to show how you can use regex correctly for exact matches within --sim.limit.

I definitely think this is useful, for now, we've added this functionality to the eest/consume-* simulators via some helper syntax in --sim.limit: The id: prefix tells --sim.limit to only run the single test case (it still does a regex match, but escapes the provided test case id for the user):

--sim.limit "id:tests/prague/eip2935_historical_block_hashes_from_state/test_block_hashes.py::test_block_hashes_history[fork_Prague-blockchain_test-full_history_plus_one_check_blockhash_first]"

So far we've had positive feedback from devs using this. Having consistent behavior across simulators with --sim.exactmatch would be preferable though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants