Skip to content

Implement end-to-end testing. #92

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jun 16, 2023

Conversation

ehmatthes
Copy link
Contributor

As discussed in #55, implements a small set of end-to-end tests.

  • To run tests, you'll need to run pip install pytest if you don't already have it available.
  • I've run the tests on macOS and Windows. I believe they should run on most Linux systems, but have not checked that yet.
  • The image files that are generated on Windows are significantly different (pixel-wise) than what's generated on macOS. This looks like a font-rendering difference. To address this, there are Windows-specific reference files in the test resources.
  • I don't know if this is a Windows-specific issue, or something particular to my Windows setup. We may end up needing to have a small variety of reference images, and only fail a test if none of the reference images match.
  • Includes brief documentation about running tests, and adding new tests.
  • You can make tests fail by adding a call to run_git_reset(tmp_repo) in any of the test functions. This is helpful for seeing what test failures look like, and to make sure the image comparison function can actually fail appropriately.
  • The test functions are written so that every test function contains the raw command that we would expect users to run, such as git-sim log. A helper function then inserts the global options needed to make the tests work.

Eric Matthes added 3 commits June 13, 2023 20:48
Signed-off-by: Eric Matthes <eric@Erics-Mac-Studio.local>
Signed-off-by: Eric Matthes <eric@Erics-Mac-Studio.local>
Signed-off-by: Eric Matthes <eric@Erics-Mac-Studio.local>
@ehmatthes
Copy link
Contributor Author

11 files changed is better than 92. :)

I didn't see anything to change in the documentation; we still end up with a temp folder called sample_repo, which pytest renames to sample_repo0.

If you end up merging this, I think you should add a couple tests for commands you'd like to see covered, and ask any questions you have about working with the test suite. I don't have any ambitions to add a bunch of tests; my goal was to help get a test suite off the ground. If you want help with testing I'm happy to do more, but I think the bigger priority is that you're comfortable working with what's started here.

I would note that while using git-dummy is perfectly reasonable and understandable, I don't think it's as fast as copying a static repo. I think it might be reasonable to have a git-dummy call per test module, but I don't think you'd want a separate call per test function. I think the git-dummy call took about 2 seconds on my system, but I didn't benchmark that at all. I just noticed it went from ~6.3s for the test run with the static repo to ~8.3s using git-dummy. That would add up fast. You can start to see the impact by changing the scope of the tmp_repo() fixture from session to function. That runs the fixture once per test function rather than once for the whole session. For me, that's 8.3s -> 10.0s.

It's all good, you'll figure out how much test coverage is helpful to have without slowing down your development process. :)

@initialcommit-io initialcommit-io merged commit 0a9b2d3 into initialcommit-com:dev Jun 16, 2023
@ehmatthes ehmatthes mentioned this pull request Jun 16, 2023
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants