Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Query tests #184

Merged
merged 8 commits into from
Jul 31, 2020
Merged

Query tests #184

merged 8 commits into from
Jul 31, 2020

Conversation

greg-szabo
Copy link
Member

Description

This is a first stab at testing the queries in the relayer-cli. It uses abscissa's acceptance testing method.

This method is hard to troubleshoot and configuration is not granular enough. It works fine, so we should use it until better methods are implemented.

The relevant issues are already closed (#130, cosmos/ibc-rs#130, #151) but testing was missing there.


For contributor use:

  • Unit tests written
  • Added test to CI if applicable
  • Updated CHANGELOG_PENDING.md
  • Linked to Github issue with discussion and accepted design OR link to spec that describes this work.
  • Updated relevant documentation (docs/) and code comments
  • Re-reviewed Files changed in the Github PR explorer

Copy link
Member

@romac romac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not quite sure I understand the CI config changes, but the tests otherwise look good to me 👍

By the way, here's a quick and dirty solution for handling both local colored output and CI plain output so that one can run the locally as well: https://docs.rs/strip-ansi-escapes/0.1.0/strip_ansi_escapes/

We can deal properly with colors in a future PR.

@greg-szabo
Copy link
Member Author

CI config changes:
Previously, we had a chain_a and chain_b service set up and the realyer-cli called with the relayer_conf_example.toml configuration. This is the setup described in the main README of the repo. The test was just to start the relayer successfully.

This previous setup is not very useful because as it turns out gaiad (the implementation of chain_a and chain_b) is not ready for any kind of IBC testing.

This new setup comments out the currently unusable test. In the future (when gaiad is working again), we should do integration testing with two chains and this code will be relevant again. I can remove it to keep things clearer, but we'll definitely readd this in the future.

The new setup establishes the use of the simapp, which is the Cosmos-SDK testing tool. This tool has enough implementation details now to be useful for query testing. We cannot test transactions with it, though, so there's no point in settin g up two of it. Simapp has a different configuration (chain name, etc), so we use the appropriate simple_config.toml configuration with it - or in this testing case, the simd_config() function will set up the relayer to use the details of the simd setup.

In the future, the current simd-based testing should go away and the currently commented out chain_a/chain_b testing should return.

@greg-szabo
Copy link
Member Author

I looked into using trip_ansi_escapes with the tests but I'm running into interesting stuff: Abscissa has it's internal method to assert the standard out. I give it the string it should be and it internally compares it to the stdout of the application. Which means I have no means of stripping the stdout of colors, if I want to use the internal expect_line or expect_regex methods.

I should be able to extract the stdout from the result and manually strip it and parse it. All I'm losing is the nice feature where I hand over a Vector of strings and it automatically parses items as lines of the stdout, but I should be able to reimplement that (or parse out the first line, which is all we need.)

I feel we're trying to apply a quick and dirty solution that guts out some of the Abscissa features which is the main point of using Abscissa testing. I'd rather hold on with this and implement the currently agreed method of leaving out Abscissa and command-line parsing from testing.
After that's implemented we can compare and contemplate what method would be better. Then, I'd be happy to build a color-neutral version of the parser or apply this strip method to the stdout.

How does that sound? If you feel strongly about applying trip_ansi_escapes, I'll work on it, but maybe we can gain more by postponing it to a later step.

@greg-szabo
Copy link
Member Author

I've implemented the three tests without abscissa too. Based on my experience we should remove the abscissa tests, they are not needed and the new tests are more robust.

As a next step, I would like to decompose the "simd" service into some mocked chain, but I need some more information on that. I'm not sure if that should go into this PR or a separate one.

@romac
Copy link
Member

romac commented Jul 31, 2020

How does that sound? If you feel strongly about applying strip_ansi_escapes, I'll work on it, but maybe we can gain more by postponing it to a later step.

Sure, that sounds good! I was just mentioning as a potential quick and dirty hack, but didn't realize it wouldn't even work in that case.

@romac romac self-requested a review July 31, 2020 10:42
@romac
Copy link
Member

romac commented Jul 31, 2020

This new setup comments out the currently unusable test. In the future (when gaiad is working again), we should do integration testing with two chains and this code will be relevant again. I can remove it to keep things clearer, but we'll definitely readd this in the future.

Thanks for the explanation! I am fine with leaving the commented out config, but we should perhaps add a comment there explaining why it's commented out.

@greg-szabo greg-szabo changed the title Query tests using abscissa Query tests Jul 31, 2020
@greg-szabo
Copy link
Member Author

greg-szabo commented Jul 31, 2020

Good point about the CI. I looked through it again and realized that it's not worth keeping it.

In this last commit, I

  • Removed CI commented out code
  • Removed Abscissa query tests - they can't handle color/non-color output in a seamless fashion
  • Moved the latest query tests into an integration.rs for better code arrangement

Copy link
Member

@romac romac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Much more readable, but also much more robust since we are not testing against the output of the CLI, which will likely change substantially in the future. Great job :)

@romac romac merged commit 9482fd2 into master Jul 31, 2020
@romac romac deleted the greg/query-testing branch July 31, 2020 14:07
hu55a1n1 pushed a commit to hu55a1n1/hermes that referenced this pull request Sep 13, 2022
* Query tests using abscissa

* Old integration test disabled for now

* Sad typo fix

* Disabled color on acceptance tests

* Retarget integration tests to pull_requests

* Added query tests without using abscissa

* Removed Abscissa testing, cleaned up CI folder

* fmt sad fix
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants