Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate drop in API test coverage, which results in build check failures in PRs, and come up with next steps #1342

Open
3 tasks
ccostino opened this issue Sep 27, 2024 · 0 comments
Labels
engineering python Pull requests that update Python code

Comments

@ccostino
Copy link
Contributor

ccostino commented Sep 27, 2024

We recently had a PR fail the build check step due to a drop in test coverage, which you can see here.

In this case, the failure happened in PR that was a change to a GitHub Action YAML configuration file, which is not a part of the application code itself and therefore not something I'd expect to be the cause of seeing the coverage percentage drop below our threshold. I've noticed us running into this off and on in the last few months, and it's only going to continue to hamstring us until we take care of it.

This is the first step: let's get a clear picture of where we're at and where the biggest gains lie in making a real improvement here. We should figure out if we need to adjust our coverage configuration at all (without lowering the threshold further!), and which tests we need to add/modify/eliminate that will yield the most gains for us. We don't need to cover every single gap; let's find where the biggest ones are that would help us the most.

Implementation Sketch and Acceptance Criteria

  • Review the code coverage output after running tests in the API (you can look at the build check failure here or run the tests locally to see) and identify spots where code coverage is not at or near 100%
  • Review the configuration of the coverage report itself and see if it's including things that it probably shouldn't (e.g., test files!)
  • Write up findings and recommendations in this issue so that we can we can create some follow-on stories for any work that needs to happen as a follow-up
    • Coverage configuration changes
    • Writing new tests for areas that lack full coverage (I see several files that are closer to 80% - 90% coverage instead of at or near 100%; some much lower!)
    • Eliminating tests that aren't providing us any real value

Security Considerations

  • No sensitive information should be present in any of the tests or their configuration
  • Any fake/dummy data should not include real information
  • Any reference to phone numbers must use the two AWS simulated phone numbers we now have in place
@ccostino ccostino added engineering python Pull requests that update Python code labels Sep 27, 2024
@ccostino ccostino changed the title Investigate drop in API test coverage, which results in build check failures in PRs Investigate drop in API test coverage, which results in build check failures in PRs, and come up with next steps Sep 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
engineering python Pull requests that update Python code
Projects
Status: Up-Next
Development

No branches or pull requests

1 participant