-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
driver, runner and CI for interoperable test-vector based testing. #3081
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
raulk
force-pushed
the
conformance-tests
branch
from
August 15, 2020 23:09
40eefd4
to
ed33579
Compare
This commit introduces a new package `conformance` containing: 1. the test driver to exercise Lotus against interoperable test vectors, and 2. the test runner, which integrates go test with the test vector corpus hosted at https://github.com/filecoin-project/conformance-vectors. The corpus is mounted via a git submodule. Right now, only message-class test vectors are supported. In the next week, this support will be extended to tipset-class, chain-class, and block sequence-class vectors.
raulk
force-pushed
the
conformance-tests
branch
from
August 15, 2020 23:16
ed33579
to
322b331
Compare
raulk
force-pushed
the
conformance-tests
branch
from
August 16, 2020 21:39
9123bc0
to
58c340b
Compare
raulk
force-pushed
the
conformance-tests
branch
from
August 16, 2020 21:50
58c340b
to
f1c4527
Compare
raulk
changed the title
introduce interoperable vector-based conformance testing.
driver, runner and CI for interoperable test-vector based testing.
Aug 16, 2020
This was referenced Aug 18, 2020
This was referenced Aug 18, 2020
willscott
reviewed
Aug 19, 2020
This was referenced Aug 24, 2020
Closed
whyrusleeping
approved these changes
Aug 26, 2020
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, i'd prefer minimal dependencies, but not a huge deal.
magik6k
reviewed
Aug 26, 2020
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just 1 thing to update
raulk
force-pushed
the
conformance-tests
branch
from
August 26, 2020 20:50
10bda4f
to
75e0387
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit introduces a new package
conformance
containing:corpus
package.It also modifies the CircleCI configuration to add this as a separate suite, at least for now. This work is still stabilising, and some things may break. We don't want to add noise to the current suites.
The test vector corpus (https://github.com/filecoin-project/test-vectors/ repo) is mounted onto the Lotus repo via a git submodule.
Right now, only message-class test vectors are supported. In the next week, support will be extended to tipset-class, chain-class, and block sequence-class vectors.
Alerting on consensus-breaking changes
We are still pre-mainnet and we can afford to perform frequent network resets to introduce consensus-breaking changes (e.g. actor state data model or state manipulations). However, that luxury has an expiry date, and we need to level up our game in detecting breakages that can lead to forks.
Furthermore, other teams need to test their interoperability with us, in a gradual and targeted manner.
These test vectors are generated from the reference Filecoin implementation (Lotus), at a particular version. As such, they will alert us about regressions and consensus-breaking changes, such as: changes in state data model, changes to state transformations, changes in gas pricing, changes in exit codes, etc.
In some cases, the change might be intended; in others, it might not. That's the value of this: it will force a reflection/coordination point.
TODO