Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: use pytest to collect and execute test fillers #116

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
193 commits
Select commit Hold shift + click to select a range
66a4487
feat: configure pytest as framework in project config
danceratopz Apr 26, 2023
9a628b3
feat: enable test filler execution via pytest
danceratopz Apr 26, 2023
784b709
feat: convert example fillers to pytest format
danceratopz Apr 26, 2023
9fb2dfa
fillers: port chain_id to pytest format
danceratopz Apr 26, 2023
e80d312
fillers: port eip3651 fillers to pytest format
danceratopz Apr 28, 2023
f74c895
fillers: add strict test for warm coinbase out of gas
danceratopz Apr 28, 2023
9e17bbb
style: linting remove unused imports
danceratopz Apr 28, 2023
6185126
docs: fix incorrect docstring
danceratopz Apr 28, 2023
b7f029e
fillers: remove unused test parameter id function
danceratopz Apr 28, 2023
0c19547
fillers: port push0/eip3855 fillers to pytest format
danceratopz Apr 28, 2023
404c3d8
pytest: add latest-fork parameter
marioevz May 3, 2023
2a8645f
tools: add a custom exception InvalidForkError
danceratopz May 7, 2023
17a9db1
pytest: exit gracefully if an invalid fork is specified to --invalid-…
danceratopz May 7, 2023
204c48d
pytest: add a line to pytest's header informing about latest fork config
danceratopz May 7, 2023
88d638a
tools: BlockchainTestFiller, StateTestFiller
marioevz May 5, 2023
fd7aa93
pytest: autofillable state_test, blockchain_test fixtures
marioevz May 5, 2023
69aad82
pytest: implement session fixture collector
marioevz May 5, 2023
d65aae9
pytest: minor nit
marioevz May 5, 2023
d1c818e
pytest: convert the test fixture collector to a pytest yield fixture
danceratopz May 7, 2023
1b58f40
pytest: fix output directory path in test fixture collector
danceratopz May 7, 2023
e7a0c52
pytest: move test spec wrapper class definition to the fixtures
danceratopz May 7, 2023
6f7b73e
pytest: use new version of state_test fixture in converted fillers
danceratopz May 7, 2023
2c324f5
pytest: add all converted fillers to pytest testpaths
danceratopz May 7, 2023
a0f41de
pytest: collect and write fixtures per module instead of per session
danceratopz May 7, 2023
3cd5426
pytest: add partially add ref spec
marioevz May 8, 2023
8abc51e
tools/spec: implement pytest_parameter_name for all filler types
marioevz May 8, 2023
55633dc
pytest: check test contains at least one filler type
marioevz May 8, 2023
9b59126
refactor: move latest-fork functionality to its own pytest plugin
danceratopz May 9, 2023
a145392
pytest: register state_test and blockchain_test markers in ini options
danceratopz May 7, 2023
d62a894
pytest: dynamically apply state_test or blockchain_test markers to tests
danceratopz May 7, 2023
201ea88
pytest: raise if a filler implements both a state and blockchain test
danceratopz May 7, 2023
9fa5ba8
pytest: fix marker description string
danceratopz May 7, 2023
71a16c0
pytest: configuration files change
marioevz May 11, 2023
cfb145a
pytest: Attempt to fix tox
marioevz May 11, 2023
3c40efb
chore: fix fname8 spelling complaints
danceratopz May 12, 2023
20fa624
chore: fix packaging by including sub-packages using `find_packages`
danceratopz May 12, 2023
2420b29
chore: add the py.typed marker to show ethereum_test_forks supports t…
danceratopz May 12, 2023
b3166d2
chore: make pytest_plugins a package by adding __init__ file
danceratopz May 12, 2023
dd6b1b0
chore: temporarily re-add filling.decorators to fix mypy errors
danceratopz May 12, 2023
96c05cf
chore: enable flake8 check in tox
danceratopz May 12, 2023
b16c6bd
chore: make pytest run fillers by default, not framework tests
danceratopz May 12, 2023
a26e9b8
tests: remove outdated name keyword arguments to fill_test
danceratopz May 12, 2023
c9995aa
tests: adjust/add fixture keys to new style
danceratopz May 12, 2023
e1ec19a
chore: remove irrelevant and unknown setting from pytest ini
danceratopz May 12, 2023
e511628
chore: add vscode settings that add whitelist & configure pytest
danceratopz May 12, 2023
34e0d86
chore: add a list of recommended vscode extensions
danceratopz May 12, 2023
bb8a37b
chore: add more useful vs code project settings
danceratopz May 12, 2023
0ca28f8
fillers: convert 3860 fillers to pytest format
danceratopz May 10, 2023
749bfbe
fillers: fix typehint
danceratopz May 16, 2023
6be3e71
fillers: fix StateTestFiller types/typehints
danceratopz May 16, 2023
37152ab
refactor: parametrize fillers via Op opcode instead of string
danceratopz May 16, 2023
639af48
refactor: replace Yul code with opcodes using Op
danceratopz May 16, 2023
2a1181d
chore: add 3860 fillers to testpaths in pytest.ini
danceratopz May 17, 2023
3cf5658
pytest: port eip spec version checking to a pytest plugin
danceratopz May 16, 2023
0c8ee04
fillers/eip3651: Rebase amendment
marioevz May 25, 2023
1add703
fillers: minimal port of dup filler to pytest format
danceratopz May 17, 2023
76819dd
fillers: port selfdestruct balance bug filler to pytest
danceratopz May 17, 2023
2367a40
fillers: port withdrawal fillers to pytest format
danceratopz May 17, 2023
6e30d8b
chore: update whitelist for pytest
danceratopz May 17, 2023
9b4e72a
chore: add all fillers to pytest testpaths; just exclude 4844
danceratopz May 17, 2023
7d7aafc
style: remove unused and now defunct global variable
danceratopz May 23, 2023
197a70a
Update fillers/withdrawals/withdrawals.py
marioevz May 25, 2023
9b3a1fa
fillers: change test_zero_amount from test class to test function
danceratopz Jun 1, 2023
9661407
fillers: remove global, write storage set code in place
danceratopz Jun 1, 2023
5b46f05
fillers: use new StorageDictType in dup test
danceratopz Jun 1, 2023
301cbcd
Merge pull request #14 from danceratopz/fillers/convert-remaining-fil…
danceratopz Jun 1, 2023
09445ea
chore: add docs extras section for doc build requirements
danceratopz May 24, 2023
222da1e
chore: add mkdocs build to tox test environments
danceratopz May 24, 2023
52040ae
chore: enable mkddocstrings plugin for framework/filler doc
danceratopz May 24, 2023
5235e63
chore: add mkdocs search plugin
danceratopz May 24, 2023
f6030bc
docs: add dev docs section with doc gen page
danceratopz May 24, 2023
769aeee
chore: use material theme for mkdocs; add logos
danceratopz May 24, 2023
b5cbb0d
chore: add mkdocs 'site' build dir to gitignore
danceratopz May 24, 2023
185f2b4
chore: add material-mkdocs recommended markdown_extensions config
danceratopz May 24, 2023
ad41919
chore: enable mermaid diagrams in python markdown extensions
danceratopz May 24, 2023
ad98f23
docs: add a new doc home page with overview flow diagram
danceratopz May 24, 2023
b7f7204
chore: enable mkdocs git author contribution display
danceratopz May 24, 2023
eebee6f
chore: add comment on mkdocs versioning for future use
danceratopz May 24, 2023
da97dcd
chore: fix doc build warnings to invalid cross-references
danceratopz May 24, 2023
9c7e962
chore: add social card plugin for mkdocs
danceratopz May 24, 2023
9c65ea6
docs: automatically include doc for all test fillers
danceratopz May 25, 2023
2a7e809
chore: update whitelist for docs
danceratopz May 25, 2023
1d455e8
chore: enable index docs on top-level nav pages in material
danceratopz May 25, 2023
c99b124
chore: add navigation tabs to docs via material
danceratopz May 25, 2023
1cc8199
chore: clean-up docs bad top-level navigation behaviour
danceratopz May 25, 2023
8f857a7
chore: enable material nav instant for single page-like docs
danceratopz May 25, 2023
d666a9d
docs: update library reference overview
danceratopz May 25, 2023
3c2a1e2
chore: add version spec for mkdocs-gen-files to setup
danceratopz May 25, 2023
5a1c1a4
docs: elaborate on t8n and b11r in overview and chart
danceratopz May 25, 2023
9dd7c05
docs: explain mkdocs plugins used in doc flow
danceratopz May 25, 2023
caa19ec
docs: add an init file giving a top-level of the fillers
danceratopz May 25, 2023
d9f95c5
docs: add a top-level docstring for fillers/
danceratopz May 25, 2023
2c324e3
chore: mkdocs serve will also watch ./src directory
danceratopz May 25, 2023
d231110
chore: add a filter to mkdocstrings to ignore globals
danceratopz May 25, 2023
9a60433
chore: customize ordering of objects in mkdocstrings
danceratopz May 25, 2023
91936c7
chore: disable filler doc auto-generation via env var
danceratopz May 26, 2023
d3f2dd3
docs: add a better top-level filler doc nav entry
danceratopz May 26, 2023
c2be038
docs: add refspec checking to mermaid overview chart
danceratopz May 26, 2023
1cf5832
docs: add an ascii art dir tree to repo overview
danceratopz May 26, 2023
930a6d8
docs: re-structure getting started; update to pytest
danceratopz May 26, 2023
f9ba2d9
docs: fix a few typos; add .pages nav file for tutorials
danceratopz May 26, 2023
a03333c
docs: add a screenshot for vs code testing tab
danceratopz May 26, 2023
bb0d840
docs: re-work getting started; add writing tests
danceratopz May 29, 2023
4b44401
chore: update readthedocs requirements for mkdocs-material etc
danceratopz May 29, 2023
cf26931
docs: improve getting started and writing tests
danceratopz May 29, 2023
a7dae2d
docs: minor improvements to g started & writing tests
danceratopz May 29, 2023
a67bcf9
docs: increase front page image size
danceratopz May 29, 2023
6e033c0
docs: update/remove tutorial content
danceratopz May 29, 2023
2c50d5f
docs: minor improvements to g started & writing tests
danceratopz May 29, 2023
abd3d80
docs: add a getting help section
danceratopz May 29, 2023
dee9db7
chore: add a delay after generating src doc; hack
danceratopz May 29, 2023
24aa07b
chore: disable doc build in tox py3 profile
danceratopz May 29, 2023
2a8c9d7
doc flow: fix automatic filler doc generation
danceratopz May 31, 2023
b7456f3
docs: improve ref spec docs and fix a few typos
danceratopz May 31, 2023
baa7b11
chore: re-enable doc build in tox py3 profile
danceratopz May 31, 2023
e7bc437
docs: add a panda pic to getting help
danceratopz May 31, 2023
307f854
docs: add all maintainer discord handles to getting help
danceratopz May 31, 2023
d34295d
docs: make discord handles copy-able via code field
danceratopz May 31, 2023
5c0eb15
docs: Update readme and couple of minor fixes
danceratopz May 31, 2023
36db898
docs: fill in year and holder placeholders in license
danceratopz May 31, 2023
20237f7
docs: fix image links; add contributing and license
danceratopz May 31, 2023
04e16ef
docs: fix typo in readme
danceratopz May 31, 2023
8c0a189
chore: remove now unused awesome-pages .pages files
danceratopz Jun 1, 2023
4ea0268
docs: fix link to testing new features, vs code
danceratopz Jun 1, 2023
2a69e6d
Merge pull request #16 from danceratopz/chore/add-mkdocs-to-setup
danceratopz Jun 1, 2023
8385b2a
fillers/eip-4844: update blob tx tests for pytest (#7)
marioevz Jun 1, 2023
6240140
fillers/eip-4844: pytest for excess data gas tests (#9)
marioevz Jun 1, 2023
008957d
fillers/eip-4844: Update Point Evaluation Precompile Tests for pytes…
marioevz Jun 1, 2023
709cd4d
fillers/eips/eip4844/blobhash_opcode_contexts.py: Start pytest port f…
spencer-tb May 19, 2023
19e7199
fillers/eips/eip4844/blobhash_opcode_contexts.py: Add all contexts.
spencer-tb May 19, 2023
2c5bbb8
fillers/eips/eip4844/blobhash_opcode_contexts.py: Make it more pytest…
spencer-tb May 23, 2023
fcad4fa
fillers/eip4844/opcode_contexts: Refactor with util file.
spencer-tb May 25, 2023
df5c4c5
fillers/eip4844/opcode_contexts: Add comments & small tweaks.
spencer-tb May 26, 2023
859dd04
fillers/eip4844/opcode_contexts: Update after call with Dan.
spencer-tb Jun 1, 2023
d0fbe5d
fillers/eip4844/blobhash_opcode: Port over more tests.
spencer-tb Jun 2, 2023
fea6666
fillers/eip4844/blobhash_opcode_util: Add junk to util function.
spencer-tb Jun 2, 2023
f6f1b7f
fillers/eip4844/blobhash_util: Rename util file.
spencer-tb Jun 2, 2023
6aea674
fillers/eip4844/blobhash: Update to use util function.
spencer-tb Jun 2, 2023
e60c0e5
fillers/eip4844/blobhash_util: Quick text fix.
spencer-tb Jun 2, 2023
91362c6
fillers/eip4844/blobhash_opcode: Final changes.
spencer-tb Jun 2, 2023
f78c1cc
fillers/eip4844/blobhash_opcode_contexts: Remove excessive BLOBHASH.
spencer-tb Jun 2, 2023
fc256d1
fillers/eip4844/blobhash_opcode: Line clean up.
spencer-tb Jun 2, 2023
39bdffc
fillers/eip4844/blobhash_opcode: Pre-emptive validFrom(Cancun).
spencer-tb Jun 2, 2023
d6ef0d2
fillers/point_eval_precompile.py: Update y/z vectors to be big endian…
spencer-tb May 15, 2023
caace59
fillers/point_eval_precompile.py: Fix y/z vectors to be big endian in…
spencer-tb May 15, 2023
e56f14e
fillers/point_eval_precompile.py: Update script to take endian input.
spencer-tb May 30, 2023
8361c93
Minor changes/refactoring.
spencer-tb Jun 6, 2023
00bf2b0
Run with blobhash with pytest, update excessDataGas test case to use …
spencer-tb Jun 6, 2023
fb0c18b
Run with blobhash with pytest, update excessDataGas test case to use …
spencer-tb Jun 6, 2023
9d3d749
Delete old blobhash tests.
spencer-tb Jun 6, 2023
7b77648
Rename blobhash utility file.
spencer-tb Jun 6, 2023
fac62b8
fillers/4844: update precompile vectors from source
marioevz Jun 6, 2023
939cb69
src/helpers: proper typing on add_kzg_version
marioevz Jun 6, 2023
10e43aa
fillers/eip4844: tox fixes
marioevz Jun 6, 2023
9268288
pytest: add 4844 folder
marioevz Jun 6, 2023
351c984
Merge pull request #2 from marioevz/pytest-eip-4844-blobhash-tests
spencer-tb Jun 6, 2023
84428dd
Update/fix test_blobhash_scenarios.
spencer-tb Jun 7, 2023
81ef102
Merge branch 'pytest-eip-4844-blobhash-tests' of https://github.com/s…
spencer-tb Jun 7, 2023
471f464
Update test_blobhash_gas_cost.
spencer-tb Jun 7, 2023
a5b66e7
Formatting.
spencer-tb Jun 7, 2023
dad5837
Merge pull request #15 from spencer-tb/pytest-eip-4844-blobhash-tests
danceratopz Jun 7, 2023
80a18a9
pytest: use fixtures instead of config.options for evm_bin
danceratopz Jun 2, 2023
8899a0f
pytest: add an alias/entry point for pytest called 'fill'
danceratopz Jun 2, 2023
62c1097
ethereum_test_forks: expose latest_fork_resolver in init
danceratopz Jun 2, 2023
ce6cd2e
evm_transition_tool: add a rough implementation of is_fork_supported()
danceratopz Jun 2, 2023
b498014
pytest: enable fork spec via command-line and fork validity markers
danceratopz Jun 2, 2023
82b777a
chore: update whitelist for pytest
danceratopz Jun 2, 2023
d7ba2ba
fillers/pytest: apply new validity markers to all tests
danceratopz Jun 2, 2023
2747840
pytest: move main conftest.py to src/pytest_plugins/
danceratopz Jun 2, 2023
52d31e8
pytest: fix packages to ignore for ref spec checking
danceratopz Jun 2, 2023
ac9873e
pytest: change default output directory to 'fixtures'
danceratopz Jun 2, 2023
090121a
pytest: report evm and solc versions in pytest console header
danceratopz Jun 2, 2023
0c7fd5c
pytest: fix console output 'fillers' -> 'tests'
danceratopz Jun 2, 2023
b918b8a
src/forks: add necessary transition fork helpers
marioevz Jun 2, 2023
f176500
pytest: add fork_transition_test marker
marioevz Jun 2, 2023
fea2578
fillers: use fork_transition_test marker
marioevz Jun 2, 2023
396649e
fillers: fix minor typo in comments.
danceratopz Jun 6, 2023
b22d64c
pytest: rename fork_transition_test to valid_at_transition_to
danceratopz Jun 6, 2023
9ba4579
style: fix long lines for flake8 checks
danceratopz Jun 6, 2023
05669ec
pytest: move marker definitions from pytest.ini to plugins
danceratopz Jun 6, 2023
5cab759
pytest: move plugins to their own sub-directories
danceratopz Jun 7, 2023
90bf90c
pytest: remove latest_fork plugin; superseded by forks
danceratopz Jun 7, 2023
63e416e
chore: remove unused tf-related code; minor fix in docs
danceratopz Jun 7, 2023
609c738
tools: move upcoming forks to forks; add fork getter helpers
danceratopz Jun 7, 2023
c8659d8
refactor: rename all_transition_forks() to get_transition_forks()
danceratopz Jun 7, 2023
3045500
tests: test new ethereum_test_forks helper functions
danceratopz Jun 7, 2023
695214b
docs: add docstrings to pytest plugin inits & corresponding md stubs
danceratopz Jun 7, 2023
9b579ce
tests: add some tests for the forks pytest plugin
danceratopz Jun 7, 2023
5e24572
fillers/eip4844: fix markers on blobhash tests
marioevz Jun 7, 2023
f71e536
fillers,docs: rename all test files to `test_*.py`
marioevz Jun 7, 2023
38b5f18
test_filler: strip `test_` from output fixtures
marioevz Jun 7, 2023
2a3e28e
refactor: rename 'fillers' directory to 'tests'
danceratopz Jun 7, 2023
833d5b0
Merge pull request #18 from danceratopz/pytest/add-pytest-markers-spe…
danceratopz Jun 7, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -42,4 +42,4 @@ jobs:
- name: Install Tox and any other packages
run: pip install tox requests
- name: Run Tox (CPython)
run: tox -e py3
run: tox
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ __pycache__/
*$py.class
venv/
/fixtures/
/out/

# C extensions
*.so
Expand Down Expand Up @@ -53,4 +54,8 @@ pip-delete-this-directory.txt
verify_kzg_proof

# docs
.cache
_readthedocs
site
venv-docs/
docs/_auto_gen_fillers/
16 changes: 16 additions & 0 deletions .vscode/extensions.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
{
// See https://go.microsoft.com/fwlink/?LinkId=827846 to learn about workspace recommendations.
// Extension identifier format: ${publisher}.${name}. Example: vscode.csharp
// List of extensions which should be recommended for users of this workspace.
"recommendations": [
"ms-python.python",
"ms-python.isort",
"ms-python.flake8",
"ms-python.black-formatter",
"esbenp.prettier-vscode",
"streetsidesoftware.code-spell-checker",
"tamasfe.even-better-toml",
],
// List of extensions recommended by VS Code that should not be recommended for users of this workspace.
"unwantedRecommendations": []
}
37 changes: 37 additions & 0 deletions .vscode/settings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
{
"cSpell.customDictionaries": {
"project-words": {
"name": "project-words",
"path": "${workspaceRoot}/whitelist.txt",
"description": "Words used in this project",
"addWords": true
},
"custom": true, // Enable the `custom` dictionary
"internal-terms": false // Disable the `internal-terms` dictionary
},
"[python]": {
"editor.rulers": [80],
"editor.formatOnSave": true,
"editor.defaultFormatter": "ms-python.black-formatter",
"editor.codeActionsOnSave": {
"source.organizeImports": true
}
},
"python.linting.mypyEnabled": true,
"python.linting.flake8Enabled": true,
"python.testing.promptToConfigure": false,
"python.testing.unittestEnabled": false,
"python.testing.pytestEnabled": true,
"python.testing.pytestArgs": [
// To discover tests from an upcomming fork, use the `latest-fork` argument as following.
// "--latest-fork=Cancun",
// Hopefully vscode-python will support multiple test framework environments sooon, see
// https://github.com/microsoft/vscode-python/issues/12075
// For now, to toggle between running "framework tests" and "filling tests" comment/
// uncomment the relevant line below.
"-c",
// "pytest-framework.ini",
"pytest.ini"
// "-vv"
]
}
2 changes: 1 addition & 1 deletion LICENSE
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
Copyright <YEAR> <COPYRIGHT HOLDER>
Copyright 2023 Ethereum Community

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

Expand Down
Loading