Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pytest: port eip spec version checking to a pytest plugin #12

Conversation

danceratopz
Copy link
Owner

@danceratopz danceratopz commented May 16, 2023

This PR ports the reference spec checking by implementing the functionality in a pytest plugin: spec_version_checker.

If the checks fail, they get reported as warnings in the pytest test summary.

See for example the following two runs that report warnings that:

  1. pytest was unable to check the spec version, and,
  2. the spec version is missing/notprovided, respectively, out of date.

image

There is no way from pytest's exit codes to tell whether warnings were raised. If we want to be able to check whether these warnings are present, via CI, for example, we'd have to run pytest as:

pytest -W error 

This won't run fillers though, as it's currently implemented, as the warning gets treated as an error in test setup. This will also treat all warnings in the test session as errors, with a bit more work, we could filter using our custom warnings using, for example,

pytest -W error::NoReferenceSpecDefined

In general, however, the output using -W error:: is very noisy as an error for every test is raised. The following is example output with -x (stop after first fail):
image

Unverified

This commit is not signed, but one or more authors requires that any commit attributed to them is signed.
Copy link
Collaborator

@marioevz marioevz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! I like the plug-in approach definitely.

@danceratopz
Copy link
Owner Author

Thanks @marioevz. Perhaps we could try and improve this in the future by enabling the spec checks in a separate test(?). The idea being that it could be ran as a separate check. Currently, you have to execute all the fillers in order to check that the specs are up-to-date. I think this PR adds the existing functionality from tf to pytest, though.

@danceratopz danceratopz merged commit 536a25d into feature/use-pytest-to-collect-and-hydrate-test-fillers-v2 May 17, 2023
@danceratopz danceratopz deleted the pytest/add-reference-spec-test branch June 23, 2023 06:42
danceratopz added a commit that referenced this pull request Dec 12, 2023
* feat(pytest): allow filtering by fixture format via pytest mark

* feat(fw): give unset fixture format a more verbose name
danceratopz added a commit that referenced this pull request Dec 14, 2023
…xture formats; Per-format sub-folder output (ethereum#358)

* tools: Move fixture collector to spec

* refactor: Fixtures collector

* all: remove `BaseTestConfig`

* fix: all filler tests

* docs: update

* tox: spelling

* feat: Parametrize the supported fixture types for each test type

* fix: many tests

* fix: remaining tests

* fix: fixture name case

* fix: Indirect parametrize using fixture format

* feat: Dynamic generation of spec test fixtures

* fix: type-checking

* fix: imports

* feat(pytest): allow filtering by fixture format via pytest mark (#12)

* feat(pytest): allow filtering by fixture format via pytest mark

* feat(fw): give unset fixture format a more verbose name

* feat(pytest): only paramatrize post-merge forks with hive format (#13)

* feat(pytest): remove items which are pre-merge and have hive format

* chore: make exception string clearer; fix whitespace

* fix(pytest): test_filler: remove `enable_hive` completely

* fix: Remove unused `STATE_TEST_HIVE`

---------

Co-authored-by: danceratopz <danceratopz@gmail.com>
danceratopz added a commit that referenced this pull request Jan 11, 2024
…xture formats; Per-format sub-folder output (ethereum#358)

* tools: Move fixture collector to spec

* refactor: Fixtures collector

* all: remove `BaseTestConfig`

* fix: all filler tests

* docs: update

* tox: spelling

* feat: Parametrize the supported fixture types for each test type

* fix: many tests

* fix: remaining tests

* fix: fixture name case

* fix: Indirect parametrize using fixture format

* feat: Dynamic generation of spec test fixtures

* fix: type-checking

* fix: imports

* feat(pytest): allow filtering by fixture format via pytest mark (#12)

* feat(pytest): allow filtering by fixture format via pytest mark

* feat(fw): give unset fixture format a more verbose name

* feat(pytest): only paramatrize post-merge forks with hive format (#13)

* feat(pytest): remove items which are pre-merge and have hive format

* chore: make exception string clearer; fix whitespace

* fix(pytest): test_filler: remove `enable_hive` completely

* fix: Remove unused `STATE_TEST_HIVE`

---------

Co-authored-by: danceratopz <danceratopz@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants