test files cannot contain a tests variable
#1084
Milestone
Comments
thehowl
added
🐞 bug
thehowl
added a commit
to thehowl/gno
that referenced
this issue
Nov 19, 2024
r3v4s
pushed a commit
to gnoswap-labs/gno
that referenced
this issue
Dec 10, 2024
This PR tackles a large amount of improvements in our current internal testing systems for the gnovm, as well as those for `gno test`. The primary target is the execution of filetests; however many improvements have been implemented to remove dead or duplicate code, and bring over some of the improvements that have been implemented in tests to filetests, when they come at little added "cost". The biggest headline concerns the execution of filetests. I wrote the specific improvements undertaken in a [blog post on "diary of a gnome"](https://gno.howl.moe/faster-tests/), but here's a side-by-side comparison of the execution in this PR (left) and the execution on master (right).  - Fixes gnolang#1084 - Fixes gnolang#2826 (by addressing root cause of slowness) - Fixes gnolang#588, only running `_long` filetests on master and generally speeding up test execution. ## Impact - Test context (tests and filetests) - Tests and filetests now share the same `test.Store` when running. Coupled with `test.LoadImports`, it allows us to "eagerly load" imports ahead of the test execution, and save it in the store. This is the primary performance improvement of this PR, as all pure packages can be run and preprocessed only once, whereas before it was once per package, and once per filetest. (More info in the blog post.) - One of the consequences of this is that package initialization happens outside of the test; so a filetest can no longer check the output of a `println` used in package initialization. - The default user no longer has 200 gnot by default. There are two mechanisms that make this unnecessary: `// SEND:`, and `std.TestIssueCoin`. Some test balances had to be updated. - Filetests - Running filetests in `gno test -v` now also prints their output. - Realm tests are no longer executed in `main.gno`, but in a filename following the name of the filetest; this changed some `// Realm:` directives. - The sytem to read directives and update them in the filetests has been re-written, and should now be more resilient and with fewer "exceptions". This means that leading and trailing whitespace in output now is correctly considered as output, leading to the addition of many empty `//` in the tests. - `// Error:` directives are now treated the same as everything else; and are updated if the `-update-golden-tests` flag is passed. - Removed the `imports` metric from the runtime metrics, as it's now no longer straightforward to calculate (given that imports are loaded "eagerly"). - Removed support for different "modes" of importing stdlibs; further removing support for gonative (gnolang#1361). The remaining gonative libraries are `os`, `fmt`, `encoding/json`, `internal/os_test` and `math/big`. This removes the `-with-native-fallback` flag from `gno test`. - Consequently, filetests ending with `_native.gno` have largely been removed, and those ending with `_stdlibs.go` have had their suffix removed. - Some files testing `gonative` types and functions which were only used in a small subset of these tests, have been removed. - Tests - `gno test`, for testing packages, created a `main_test.gno` file that is then called directly from the machine on each test. This creates two identifiers, `tests` and `runtest`, which could come into conflict with those defined by the tests themselves. We now call `testing.RunTest` directly, without an intermediary. - Exports in the normal tests (ie. defined in `pkg`) are now visible in the tests of `pkg_test`. This is most useful for some standard library tests, where there often is an `export_test.go` with the sole scope of exporting some internal functions and methods for testing in the "real" testing file. - `gno lint`, and other occasions where we use `issueFromError` (like when parsing errors in `gno test`), will now also print the column of the error if given. ## Summary of internal changes - pkg/gnolang - `TestFiles` is the new function running the filetests. - `eval_test.go` has been removed, moving some of its improvements over to `TestFiles`, and reducing the scope of the corresponding tests in `debugger.go`, to what it's actually supposed to test for. - As a consequence of removing all of type mappings in `imports.go`, I removed `SetStrictGo2GnoMapping` in favour of always being "strict", and not allowing defining native types of defined types any more. - The tests in `gonative_test` where primarily related to this, so I removed them. Similarly for `preprocess_test`. - `TestFunc` / `TestMemPackage` have been removed as redundant, including some of their features in `cmd/go/test.go` (like supporting exporting symbols in `_test.gno` files) - The `Store` no longer has `ClearCache` and `SetStrictGo2GnoMapping`; `ClearCache` now has a better substitute in `BeginTransaction`. - the `testing` stdlib no longer caches `matchPat` and `matchString` as pure packages should not be able to modify their values during execution, and as a result of other changes this was failing. - The tests/ directory has been removed / moved to `gnovm/pkg/test`. This directory should eventually contain the internal code for `gno test` as well; but for now I wanted to clean `tests` to ensure it is a directory just for integration tests, rather than code and testing systems. - `TestMachine` and `TestStore` have been renamed to Machine and Store, to avoid redundancy with the `test` package name. - Removed plenty instructions in `gnovm/Makefile` as now most tests are just in `pkg/gnolang`. - I removed `MsgContext.Msg` as unused. It can be re-added later if necessary. - In the CI, tests are now run with `-short`, at least until we figure out how to make the VM efficient enough to run these tests. Aside from some filetests, this now excludes some stdlibs: `bytes`, `strconv`, `regexp/syntax`. I accept other proposals that could make us have fast tests on PR's, while still testing (mostly) everything. --- [](https://lu.ma/open-source-saturday-torino) --------- Co-authored-by: Marc Vertes <marc.vertes@tendermint.com>
albttx
pushed a commit
that referenced
this issue
Jan 10, 2025
](https://lu.ma/open-source-saturday-torino){kind=link}
This PR tackles a large amount of improvements in our current internal testing systems for the gnovm, as well as those for `gno test`. The primary target is the execution of filetests; however many improvements have been implemented to remove dead or duplicate code, and bring over some of the improvements that have been implemented in tests to filetests, when they come at little added "cost". The biggest headline concerns the execution of filetests. I wrote the specific improvements undertaken in a [blog post on "diary of a gnome"](https://gno.howl.moe/faster-tests/), but here's a side-by-side comparison of the execution in this PR (left) and the execution on master (right).  - Fixes #1084 - Fixes #2826 (by addressing root cause of slowness) - Fixes #588, only running `_long` filetests on master and generally speeding up test execution. ## Impact - Test context (tests and filetests) - Tests and filetests now share the same `test.Store` when running. Coupled with `test.LoadImports`, it allows us to "eagerly load" imports ahead of the test execution, and save it in the store. This is the primary performance improvement of this PR, as all pure packages can be run and preprocessed only once, whereas before it was once per package, and once per filetest. (More info in the blog post.) - One of the consequences of this is that package initialization happens outside of the test; so a filetest can no longer check the output of a `println` used in package initialization. - The default user no longer has 200 gnot by default. There are two mechanisms that make this unnecessary: `// SEND:`, and `std.TestIssueCoin`. Some test balances had to be updated. - Filetests - Running filetests in `gno test -v` now also prints their output. - Realm tests are no longer executed in `main.gno`, but in a filename following the name of the filetest; this changed some `// Realm:` directives. - The sytem to read directives and update them in the filetests has been re-written, and should now be more resilient and with fewer "exceptions". This means that leading and trailing whitespace in output now is correctly considered as output, leading to the addition of many empty `//` in the tests. - `// Error:` directives are now treated the same as everything else; and are updated if the `-update-golden-tests` flag is passed. - Removed the `imports` metric from the runtime metrics, as it's now no longer straightforward to calculate (given that imports are loaded "eagerly"). - Removed support for different "modes" of importing stdlibs; further removing support for gonative (#1361). The remaining gonative libraries are `os`, `fmt`, `encoding/json`, `internal/os_test` and `math/big`. This removes the `-with-native-fallback` flag from `gno test`. - Consequently, filetests ending with `_native.gno` have largely been removed, and those ending with `_stdlibs.go` have had their suffix removed. - Some files testing `gonative` types and functions which were only used in a small subset of these tests, have been removed. - Tests - `gno test`, for testing packages, created a `main_test.gno` file that is then called directly from the machine on each test. This creates two identifiers, `tests` and `runtest`, which could come into conflict with those defined by the tests themselves. We now call `testing.RunTest` directly, without an intermediary. - Exports in the normal tests (ie. defined in `pkg`) are now visible in the tests of `pkg_test`. This is most useful for some standard library tests, where there often is an `export_test.go` with the sole scope of exporting some internal functions and methods for testing in the "real" testing file. - `gno lint`, and other occasions where we use `issueFromError` (like when parsing errors in `gno test`), will now also print the column of the error if given. ## Summary of internal changes - pkg/gnolang - `TestFiles` is the new function running the filetests. - `eval_test.go` has been removed, moving some of its improvements over to `TestFiles`, and reducing the scope of the corresponding tests in `debugger.go`, to what it's actually supposed to test for. - As a consequence of removing all of type mappings in `imports.go`, I removed `SetStrictGo2GnoMapping` in favour of always being "strict", and not allowing defining native types of defined types any more. - The tests in `gonative_test` where primarily related to this, so I removed them. Similarly for `preprocess_test`. - `TestFunc` / `TestMemPackage` have been removed as redundant, including some of their features in `cmd/go/test.go` (like supporting exporting symbols in `_test.gno` files) - The `Store` no longer has `ClearCache` and `SetStrictGo2GnoMapping`; `ClearCache` now has a better substitute in `BeginTransaction`. - the `testing` stdlib no longer caches `matchPat` and `matchString` as pure packages should not be able to modify their values during execution, and as a result of other changes this was failing. - The tests/ directory has been removed / moved to `gnovm/pkg/test`. This directory should eventually contain the internal code for `gno test` as well; but for now I wanted to clean `tests` to ensure it is a directory just for integration tests, rather than code and testing systems. - `TestMachine` and `TestStore` have been renamed to Machine and Store, to avoid redundancy with the `test` package name. - Removed plenty instructions in `gnovm/Makefile` as now most tests are just in `pkg/gnolang`. - I removed `MsgContext.Msg` as unused. It can be re-added later if necessary. - In the CI, tests are now run with `-short`, at least until we figure out how to make the VM efficient enough to run these tests. Aside from some filetests, this now excludes some stdlibs: `bytes`, `strconv`, `regexp/syntax`. I accept other proposals that could make us have fast tests on PR's, while still testing (mostly) everything. --- [](https://lu.ma/open-source-saturday-torino) --------- Co-authored-by: Marc Vertes <marc.vertes@tendermint.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I don't know if the issue is as specific like the title, but here is a minimal example:
This was discovered while trying to run
gno test .on a converted version of https://github.com/notnil/chess . (as a fun fact, yaegi doesn't correctly handle that package's tests either: traefik/yaegi#1585 -- looks like we found ourselves a challenge-package!)I did some debugging by adding this print statement in
gnovm/pkg/gnolang/nodes.go:The relevant output is in the last lines
I'm not 100% sure on the functioning of
Define2, but a rough guess seems to be that it is seen as a re-definition and thus panics.I suspect this issue is not limited to the
testsvariable, but this needs more investigation.(The specific issue disappears changing the name of
var teststo something else, likexxtests.)(Part of a series of issues encountered while developing gnochess)
The text was updated successfully, but these errors were encountered: