-
Notifications
You must be signed in to change notification settings - Fork 997
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add new decorators to skip tests and handle it in testgen #2016
Conversation
1. Add `with_configs` decorator to assign available configs 2. Add `only_full_crosslink` decorator to detect if the configuation can do full crosslinking 3. Add `context.is_pytest` flag: True if calling via pytest. False if calling from test generator.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
well done!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice work! Changes look good, but there are a few older tests where the test had to be skipped. Updating these to raise the new SkippedTest
would be best I think.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Sorry, 3 different parts in this PR.
Issue 1 (minor)
get_custody_secret
is duplicate.Solution for Issue 1
Use v-guide's
get_custody_secret
in test helper.Issue 2
For some test cases, it can't be passed with mainnet configs.
Solution for Issue 2
with_configs
decorator to assign available configsonly_full_crosslink
decorator to detect if the configuration cando full crosslinking: some test cases assume that each slot has enough committee to form crosslink.
CONFIG_NAME
to the config files."mainnet"
or"minimal"
.Issue 3
pytest
.pytest
to run the tests.It would be better to handle the "unavailable" forks and configs properly. e.g., some test cases are only for phase 0, or some test cases are only for mainnet configs.
Solution for Issue 3
Add
context.is_pytest
flag: True if calling viapytest
. False if calling from test generator.Add
dump_skipping_message
to dump message topytest
caller or test generator runner:pytest
: mark it isskip
. For example, if we callpytest --disable-bls --config=mainnet eth2spec/test/phase1/sanity/test_blocks.py
, it will show some test cases are skipped:SkippedTest
exception will be caught and show message like:Add
pytest
togen_helpers
requirements. But note that it's just becausepytest
is imported ineth2spec/test/context.py
, the test generator does NOT use it.