Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add txtar driver for gnoland integration test #1117

Merged
merged 49 commits into from
Oct 5, 2023

Conversation

gfanton
Copy link
Member

@gfanton gfanton commented Sep 13, 2023

Resolves #1101.

This PR establishes a base for creating txtar tests using a partial in-memory gnoland node. It supports:

  • gnoland [start|stop]: The node doesn't start automatically, which enables users to perform pre-configuration or pass custom arguments to the start command.
  • gnokey: Most of the commands should work. The --remote, --insecure-password-stdin, and --home flags are automatically set with the appropriate values to communicate with the node.
  • sleep: A simple helper to introduce a delay between actions, ensuring specific tasks complete before proceeding.

Currently, only the "test1" user is automatically created in the default keybase directory (without a password). Default gnoland genesis balance and genesis transaction files are also load on start.

You can find some examples of txtar in this directory:
https://github.com/gfanton/gno/tree/feat/gnoland-txtar-driver/gno.land/cmd/gnoland/testdata

gnoland logs aren't forwarded to stdout to avoid overwhelming informations in the tests. Instead, you can specify a log directory using the LOG_DIR environment variable or set TESTWORK=true to enable persistence in the txtar working directory. In either case, the path to the log file will be printed at the start if one of these environment variables is set.
This also enables storing logs as artifacts on the CI for later examination.

There is still a lot to do, but I believe this provides a good base for future iterations.

Contributors' checklist...
  • Added new tests, or not needed, or not feasible
  • Provided an example (e.g. screenshot) to aid review or the PR is self-explanatory
  • Updated the official documentation or not needed
  • No breaking changes were made, or a BREAKING CHANGE: xxx message was included in the description
  • Added references to related issues and PRs
  • Provided any useful hints for running manual tests
  • Added new benchmarks to generated graphs, if any. More info here.

@github-actions github-actions bot added the 📦 ⛰️ gno.land Issues or PRs gno.land package related label Sep 13, 2023
@gfanton gfanton changed the title wip: add txtar driver wip: add txtar driver for gnoland integration test Sep 13, 2023
gno.land/pkg/gnoland/app.go Outdated Show resolved Hide resolved
gno.land/pkg/gnoland/app.go Outdated Show resolved Hide resolved
Copy link
Member

@moul moul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It overall looks good, ping me again when you'll need another review.

@codecov
Copy link

codecov bot commented Sep 18, 2023

Codecov Report

Attention: 110 lines in your changes are missing coverage. Please review.

Comparison is base (ce258b1) 47.31% compared to head (57497cf) 47.59%.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1117      +/-   ##
==========================================
+ Coverage   47.31%   47.59%   +0.27%     
==========================================
  Files         367      369       +2     
  Lines       62118    62549     +431     
==========================================
+ Hits        29394    29770     +376     
- Misses      30325    30370      +45     
- Partials     2399     2409      +10     
Files Coverage Δ
gno.land/pkg/sdk/vm/keeper.go 40.10% <100.00%> (+0.16%) ⬆️
gno.land/cmd/gnokey/main.go 0.00% <0.00%> (ø)
tm2/pkg/crypto/keys/client/broadcast.go 0.00% <0.00%> (ø)
tm2/pkg/crypto/keys/client/add.go 31.08% <0.00%> (ø)
tm2/pkg/crypto/keys/client/addpkg.go 0.00% <0.00%> (ø)
tm2/pkg/crypto/keys/client/call.go 0.00% <0.00%> (ø)
tm2/pkg/crypto/keys/client/delete.go 24.67% <0.00%> (ø)
tm2/pkg/crypto/keys/client/export.go 36.79% <0.00%> (ø)
tm2/pkg/crypto/keys/client/generate.go 53.03% <0.00%> (ø)
tm2/pkg/crypto/keys/client/import.go 33.92% <0.00%> (ø)
... and 9 more

... and 9 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@moul moul mentioned this pull request Sep 20, 2023
8 tasks
@gnolang gnolang deleted a comment from netlify bot Sep 20, 2023
@github-actions github-actions bot added the 📦 🌐 tendermint v2 Issues or PRs tm2 related label Sep 21, 2023
@gfanton gfanton changed the title wip: add txtar driver for gnoland integration test feat: add txtar driver for gnoland integration test Sep 21, 2023
@gfanton gfanton marked this pull request as ready for review September 25, 2023 06:39
@gfanton gfanton requested review from jaekwon and a team as code owners September 25, 2023 06:39
@gfanton gfanton requested a review from moul September 25, 2023 06:41
** as requested, this is a squash of the following commits:
- feat: use commands.IO
- chore: cleanup gnoland output using context logger
- chore: cleanup integration test
- chore: cleanup
- feat: add test data example
- chore: cleanup
- fix: concurent run of gnoland
- chore: cleanup
- fix: cleanup
- fix: docker integration test
- chore: cleanup root dir and validate config
- fix: correctly update command io argument
- chore: cleanup

Signed-off-by: gfanton <8671905+gfanton@users.noreply.github.com>
Signed-off-by: gfanton <8671905+gfanton@users.noreply.github.com>
Signed-off-by: gfanton <8671905+gfanton@users.noreply.github.com>
@gfanton
Copy link
Member Author

gfanton commented Oct 4, 2023

@thehowl I agree that deeper documentation is needed, especially with an entrypoint from docs/. However and execpt if you feel it is really mandatory, I don't think I should address it in this PR, which is already hard to review. As a counterpart, I've documented the integration package with a doc.go to provide better clarity on its usage. wdyt ?

Signed-off-by: gfanton <8671905+gfanton@users.noreply.github.com>
Signed-off-by: gfanton <8671905+gfanton@users.noreply.github.com>
Signed-off-by: gfanton <8671905+gfanton@users.noreply.github.com>
Signed-off-by: gfanton <8671905+gfanton@users.noreply.github.com>
@gfanton
Copy link
Member Author

gfanton commented Oct 4, 2023

Ok, you got me, I made the /docs/testing_guide.md as a first draft

@gfanton gfanton requested a review from thehowl October 4, 2023 23:05
Copy link
Member

@thehowl thehowl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving to unblock. Mostly minor things :)

gno.land/pkg/gnoland/app.go Outdated Show resolved Hide resolved
gno.land/pkg/integration/testing_integration.go Outdated Show resolved Hide resolved
gno.land/pkg/integration/testing_integration.go Outdated Show resolved Hide resolved
gno.land/pkg/integration/testing_integration.go Outdated Show resolved Hide resolved
gno.land/pkg/integration/gnoland.go Outdated Show resolved Hide resolved
gno.land/pkg/integration/gnoland.go Outdated Show resolved Hide resolved
Signed-off-by: gfanton <8671905+gfanton@users.noreply.github.com>
Signed-off-by: gfanton <8671905+gfanton@users.noreply.github.com>
Signed-off-by: gfanton <8671905+gfanton@users.noreply.github.com>
Copy link
Contributor

@ajnavarro ajnavarro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code looks great, but I have hard feelings about this kind of non-standard tests.

They tend to be the only way the codebase is tested, not using unit tests, so eventually, devs are not forced to make their codebase more testable (code easy to test is also better code and easy to understand and refactor), they just have to add some of these integration tests.

An example of what I'm talking about:

https://github.com/ipfs/kubo/tree/master/test/sharness

IPFS/Kubo had thousands of this kind of integration tests, and the core team spent thousands of hours fixing them when they broke. It is not an easy task due to strange errors (from the code and from the custom framework) that require hours to be able to know what is happening. The core team is getting rid of them bit by bit now.

Non-standard ways of testing (or doing anything, really) make the process more time-consuming for both parties, the one creating the tests, and the one fixing it when it breaks.

Summoning @thehowl and @moul for feedback about this concern.

@gfanton
Copy link
Member Author

gfanton commented Oct 5, 2023

@ajnavarro I was re-thinking about this and when compared to bash script-based tests, testscripts have distinct advantages. They are integrated within the Go testing framework and run within it. Everything is sandboxed, and while the tests might looks like scripts, what's actually being executed are golang methods, not binaries. This distinction is significant because you don't have to rebuild the project or depend on external binaries, which can be error-prone. Instead, you test in the standard Go way using go test. Additionally, the features available in testscripts are strictly limited. You can't use pipes or conditions here; you can only run specific commands and helpers that you define yourself directly in Go.
Even if testscripts might deviate from traditional Go testing in appearance, their integration with the Go framework, reliability, and controlled environment can make them a powerful testing tool.
That said, this shouldn't be something systematic; standard tests should always remain the primary approach.

Copy link
Member

@zivkovicmilos zivkovicmilos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To say this is a step in the right direction is an understatement 🙂
It's more like a leap in how we verify functionality.

Having said this, I've noted elsewhere I'm not a huge fan of this type of testing, but I am also fully aware we need to test e2e flows (especially with commands). The node orchestration will come in handy for lots of other efforts.

Thank you for the detailed documentation and explanations 🙏


func (c *AppOptions) validate() error {
if c.Logger == nil {
return fmt.Errorf("no logger provided")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick: this can be a regular errors.New, since nothing is being formatted

Same nit for the other check in validate

@thehowl
Copy link
Member

thehowl commented Oct 5, 2023

The code looks great, but I have hard feelings about this kind of non-standard tests.

They tend to be the only way the codebase is tested, not using unit tests, so eventually, devs are not forced to make their codebase more testable (code easy to test is also better code and easy to understand and refactor), they just have to add some of these integration tests.

@ajnavarro I tend to agree that in general we need to have a larger testing efforts such that when we add tests, we need to add them first and foremost at the unit level, then maybe at the integration level.

That said, since we're coming from a codebase we've "inherited", unless somebody takes up the task of precisely testing and refactoring many of the components that currently don't have tests -- with experience with things we've encountered with GnoChess, I'd say one of the many that is in deep need of unit tests is GnoVM reference counting and realm persistance storage -- I don't think we can achieve effective good unit testing in the short term.

txtar will be useful and in fact has already been useful in testing complex behaviour which for our current understanding cannot be easily replicated merely with unit tests.

Taking into account your concerns, I propose that we don't consider txtar tests in our coverage reports (except maybe for gnovm/pkg/integration). This is already the case, but #1122 was due to change that. So I propose we change what we were doing in #1122 to only make the gnovm/tests tests track coverage for gnovm/pkg/gnolang.

This way when a new change is added, containing txtar tests but not unit tests, then codecov will still complain that the coverage is not good enough -- and we agree (socially) that we normally don't merge PRs if the added changes are not covered. WDYT?

@moul moul merged commit 924d62d into gnolang:master Oct 5, 2023
176 checks passed
@gfanton gfanton deleted the feat/gnoland-txtar-driver branch October 5, 2023 16:24
@tbruyelle
Copy link
Contributor

Why not establish rules for the use of testscript ? For instance:

  • you have to add gno files in a testdata folder
  • you have to add a new realm in examples/gno.land/
  • you have to run multiple transactions
  • ...?

If none of these rules are true, then you shouldn't use testscript.

@moul
Copy link
Member

moul commented Oct 9, 2023

Related with #1215.

@thehowl
Copy link
Member

thehowl commented Oct 10, 2023

@tbruyelle

Why not establish rules for the use of testscript ? For instance:

* you have to add gno files in a testdata folder

* you have to add a new realm in `examples/gno.land/`

* you have to run multiple transactions

* ...?

maybe make a PR on https://github.com/gnolang/gno/blob/master/docs/testing_guide.md to discuss?

@gfanton gfanton mentioned this pull request Oct 12, 2023
10 tasks
gfanton added a commit to gfanton/gno that referenced this pull request Nov 9, 2023
Resolves gnolang#1101.

This PR establishes a base for creating `txtar` tests using a partial
in-memory `gnoland` node. It supports:

- `gnoland [start|stop]`: The node doesn't start automatically, which
enables users to perform pre-configuration or pass custom arguments to
the start command.
- `gnokey`: Most of the commands should work. The `--remote`,
`--insecure-password-stdin`, and `--home` flags are automatically set
with the appropriate values to communicate with the node.
- `sleep`: A simple helper to introduce a delay between actions,
ensuring specific tasks complete before proceeding.

Currently, only the "test1" user is automatically created in the default
keybase directory (without a password). Default `gnoland` genesis
balance and genesis transaction files are also load on start.

You can find some examples of `txtar` in this directory:

https://github.com/gfanton/gno/tree/feat/gnoland-txtar-driver/gno.land/cmd/gnoland/testdata

`gnoland` logs aren't forwarded to stdout to avoid overwhelming
informations in the tests. Instead, you can specify a log directory
using the `LOG_DIR` environment variable or set `TESTWORK=true` to
enable persistence in the `txtar` working directory. In either case, the
path to the log file will be printed at the start if one of these
environment variables is set.
This also enables storing logs as artifacts on the CI for later
examination.

There is still a lot to do, but I believe this provides a good base for
future iterations.

<!-- please provide a detailed description of the changes made in this
pull request. -->

<details><summary>Contributors' checklist...</summary>

- [ ] Added new tests, or not needed, or not feasible
- [ ] Provided an example (e.g. screenshot) to aid review or the PR is
self-explanatory
- [ ] Updated the official documentation or not needed
- [ ] No breaking changes were made, or a `BREAKING CHANGE: xxx` message
was included in the description
- [ ] Added references to related issues and PRs
- [ ] Provided any useful hints for running manual tests
- [ ] Added new benchmarks to [generated
graphs](https://gnoland.github.io/benchmarks), if any. More info
[here](https://github.com/gnolang/gno/blob/master/.benchmarks/README.md).
</details>

---------

Signed-off-by: gfanton <8671905+gfanton@users.noreply.github.com>
Co-authored-by: Manfred Touron <94029+moul@users.noreply.github.com>
Co-authored-by: Morgan <git@howl.moe>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
📦 🌐 tendermint v2 Issues or PRs tm2 related 📦 ⛰️ gno.land Issues or PRs gno.land package related
Projects
Status: Done
Status: 🌟 Wanted for Launch
Status: Done
Archived in project
Development

Successfully merging this pull request may close these issues.

New integration test method for gnoland
6 participants