Enable test generation for phase1 #1957

protolambda · 2020-07-07T19:39:03Z

Updates SSZ static test gen to deal with phase 1 types too. Needed a minor fix in random-output generation.
Update epoch processing, block operations, sanity, rewards test gen to:
- Output for phase0 like before
- Output for phase-1 specific tests
- Run phase0 module tests, with phase 1 spec. (Incompatible tests are handled by the test decorators)
Rewards and penalties mainnet epoch processing was still disabled, partially because we have rewards tests, partially because it used to be very slow. Let's try to enable them.
Rewards tests are now split into random/basic/leak, instead of all in core. We can change that back, but it seemed more consistent
Added trace-backs to test gen errors, to make debugging easier, as the error message is sometimes empty (e.g. asserts).
Unsure about phase-1 "genesis" tests. To start immediately into a phase 1 state? Should that be a thing to test clients for?

Before merge:

Some block-operations have errors in phase 1 mode. It didn't get past attester slashing tests for me. If someone can help look into this, that would be nice.
I temporarily commented mainnet mode tests, to re-run test gen faster to check if it works. Since mainnet is much slower. Need to uncomment them.

CC @mkalinin @djrtwo @hwwhww Let me know if you like to change anything for phase 1 test generation.

…now.

mkalinin · 2020-07-08T18:46:59Z

Unsure about phase-1 "genesis" tests. To start immediately into a phase 1 state? Should that be a thing to test clients for?

It might be relevant for testnets or simulations that are starting from GENESIS_SLOT right away, i.e. with PHASE_1_GENESIS_SLOT = GENESIS_SLOT

protolambda · 2020-07-08T20:12:44Z

@mkalinin Sounds good, but we'll need the spec to define what that phase-1 genesis looks like (I see you are working on that and shared details in the discord, thanks). Right now I'm afraid a bare phase-0 genesis test that runs with phase-1 spec will break other phase-1 spec assumptions.

mkalinin · 2020-07-09T10:42:30Z

we'll need the spec to define what that phase-1 genesis looks like

Totally agree. I currently use pretty naive way to get phase-1 genesis:

genesis_state = initialize_beacon_state_from_eth1(eth1_block_hash, eth1_timestamp, deposits)
phase_1_genesis_state = upgrade_to_phase1(genesis_state)

But this approach works well in my case.

hwwhww · 2020-07-09T14:24:55Z

Some block-operations have errors in phase 1 mode. It didn't get past attester slashing tests for me. If someone can help look into this, that would be nice.

The error message, for example:

generating test vectors from tests source: eth2spec.test.phase0.block_processing.test_process_attester_slashing
Generating test: output/minimal/phase1/operations/attester_slashing/pyspec_tests/all_empty_indices
ERROR: failed to generate vector(s) for test output/minimal/phase1/operations/attester_slashing/pyspec_tests/all_empty_indices: 'NoneType' object is not iterable
Traceback (most recent call last):
  File "/Users/hwwang/.pyenv/versions/env38a/lib/python3.8/site-packages/gen_base/gen_runner.py", line 138, in run_generator
    for (name, out_kind, data) in test_case.case_fn():
TypeError: 'NoneType' object is not iterable

It's because we set these tests to with_phases([PHASE0]) since phase1 AttestationData format was different from phase0. And thanks to 0.01 bit proof of custody [depends on 256 bit custody atoms] #1889, these attester slashing tests can be enabled now. (I pushed 60f4bd2)
However, we still have some other test cases for test generators with with_phases([PHASE0]).
Also, we have some phase1 test cases that only work with minimal config since we need enough validators to form crosslink for every slot for every slot in these test cases (example).

A quick workaround is skipping the test case if test_case.case_fn() returns None in gen_runner.py.
@protolambda any thought? :)

1. Add `with_configs` decorator to assign available configs 2. Add `only_full_crosslink` decorator to detect if the configuation can do full crosslinking 3. Add `context.is_pytest` flag: True if calling via pytest. False if calling from test generator.

…OCHS_PER_CUSTODY_PERIOD`-long transition

Add new decorators to skip tests and handle it in testgen

Skip the too-slow custody tests and turn on the generators

…back to the spec function for other test cases afterwards

Add `disable_process_reveal_deadlines` decorator and `reveal_deadlines_setting` meta tag

protolambda · 2020-09-09T21:45:42Z

@hwwhww great work on all the phase 1 changes. Testing PRs were merged into this PR, and I just resolved a small merge conflict with dev. Is this ready to be merged now?

… instead of List

hwwhww

@protolambda
I pushed some commits:

Remove redundant keys in the rewards test generator.
Use Dict for phase_0_mods, phase_1_mods instead of List.
Add make lint_generators command for test generators flake8 checks. Also added it to the CI.

LGTM now. :)

hwwhww · 2020-09-18T09:22:02Z

@djrtwo

Changes to the phase 0 test vectors outputs:

[ssz] fix in random-output generation.
[rewards] Rewards tests are now split into random/basic/leak, instead of all in core.
[sanity/block] Adds a reveal_deadlines_setting field to the sanity/block meta.yaml.
[Config files]: add CONFIG_NAME meta field to distinguish between "mainnet" and "minimal" configs.

skip large_validator_set tests if too large (mainnet config)

prestonvanloon · 2020-09-19T16:45:50Z

Was it intentional to remove eth2.0-spec-tests/tests/mainnet/phase0/ssz_static?

protolambda · 2020-09-19T16:53:21Z

No, thanks for reporting, just heard from Terence too. It was meant to be disabled just for phase1 mainnet I think (and at first all mainnet tests were disabled, to testrun the changes in this PR quicker). We'll re-introduce them. The types have not changed, so you can keep running the v0.12.2 SSZ tests. Sorry for the inconvenience.

protolambda added 2 commits July 7, 2020 20:17

update block operations generator

6b1d483

Update test gen to output phase 1 test vectors. Mainnet disabled for …

3c374b1

…now.

protolambda added phase1 scope:CI/tests/pyspec labels Jul 7, 2020

The AttesterSlashing tests for phase1 can be enabled now after #1889

60f4bd2

djrtwo mentioned this pull request Jul 15, 2020

Apply strict uint64 casting #1746

Merged

hwwhww mentioned this pull request Jul 16, 2020

Rework initial shard slot (PHASE_1_FORK_SLOT) #1971

Merged

hwwhww and others added 17 commits August 11, 2020 15:24

Merge branch 'dev' into testgenphase1

3e5cada

Skip the testcases that don't support the fork or the config

52ba90d

Use v-guide's get_custody_secret

83760b1

Update testing context

6289664

1. Add `with_configs` decorator to assign available configs 2. Add `only_full_crosslink` decorator to detect if the configuation can do full crosslinking 3. Add `context.is_pytest` flag: True if calling via pytest. False if calling from test generator.

Add disable_process_reveal_deadlines decorator

7eeb754

Disable process_reveal_deadlines in the custody tests that have `EP…

42f368c

…OCHS_PER_CUSTODY_PERIOD`-long transition

Update the pytest version to >=4.4 as eth2spec

458166a

Add skipping messages to some old test cases

b4583ab

Merge pull request #2016 from ethereum/hwwhww/phase1-tests

8de7f95

Add new decorators to skip tests and handle it in testgen

Merge branch 'testgenphase1' into reveal_deadlines_setting

dc88817

Enable all tests

74a2a1c

Add decorators to diable testing with mainnet config

a0d646e

Enable more rewards and finality tests

916ae9e

Merge pull request #2038 from ethereum/testgenphase1_enable_all

e57e140

Skip the too-slow custody tests and turn on the generators

Fix disable_process_reveal_deadlines decorator: should have set it …

fd4e7dd

…back to the spec function for other test cases afterwards

Merge pull request #2017 from ethereum/reveal_deadlines_setting

62cd4e8

Add `disable_process_reveal_deadlines` decorator and `reveal_deadlines_setting` meta tag

Merge branch 'dev' into testgenphase1

28137a6

Remove redundant keys and use Dict for phase_0_mods, phase_1_mods…

c0c2461

… instead of List

Add make lint_generators linter check for test generators

20bd00c

hwwhww approved these changes Sep 15, 2020

View reviewed changes

hwwhww and others added 3 commits September 17, 2020 21:11

Merge branch 'dev' into testgenphase1

5374890

skip large_validator_set tests if too large

0c1784b

Use with_configs[MINIMAL] on large_validator_set tests

27d8c80

djrtwo added 2 commits September 18, 2020 07:59

remove kludge to skip large_validator_set on mainnet config

d257926

Merge pull request #2075 from ethereum/skip-huge-test

87220f8

skip large_validator_set tests if too large (mainnet config)

djrtwo approved these changes Sep 18, 2020

View reviewed changes

djrtwo merged commit 68bcc19 into dev Sep 18, 2020

djrtwo deleted the testgenphase1 branch September 18, 2020 14:44

terencechain mentioned this pull request Sep 19, 2020

Update spec tests to v0.12.3 prysmaticlabs/prysm#7276

Merged

protolambda mentioned this pull request Sep 19, 2020

Re-introduce missing SSZ tests (v0.12.3 test release) #2076

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable test generation for phase1 #1957

Enable test generation for phase1 #1957

protolambda commented Jul 7, 2020

mkalinin commented Jul 8, 2020

protolambda commented Jul 8, 2020

mkalinin commented Jul 9, 2020

hwwhww commented Jul 9, 2020 •

edited

Loading

protolambda commented Sep 9, 2020

hwwhww left a comment

hwwhww commented Sep 18, 2020

prestonvanloon commented Sep 19, 2020

protolambda commented Sep 19, 2020

Enable test generation for phase1 #1957

Enable test generation for phase1 #1957

Conversation

protolambda commented Jul 7, 2020

mkalinin commented Jul 8, 2020

protolambda commented Jul 8, 2020

mkalinin commented Jul 9, 2020

hwwhww commented Jul 9, 2020 • edited Loading

protolambda commented Sep 9, 2020

hwwhww left a comment

Choose a reason for hiding this comment

hwwhww commented Sep 18, 2020

prestonvanloon commented Sep 19, 2020

protolambda commented Sep 19, 2020

hwwhww commented Jul 9, 2020 •

edited

Loading