Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pytest-sentry for pytest runs #7186

Merged
merged 1 commit into from
Nov 23, 2022
Merged

Conversation

drew2a
Copy link
Contributor

@drew2a drew2a commented Nov 17, 2022

This PR helps us to collect statistics about flaky tests.

Let's say you have a testsuite with some flaky tests that randomly break your CI build due to network issues, race conditions or other stuff that you don't want to fix immediately. The known workaround is to retry those tests automatically, for example using pytest-rerunfailures.
One concern against plugins like this is that they just hide the bugs in your testsuite or even other code. After all your CI build is green and your code probably works most of the time.
pytest-sentry tries to make that choice a bit easier by tracking flaky test failures in a place separate from your build status. Sentry is already a good choice for keeping tabs on all kinds of errors, important or not, in production, so let's try to use it in testsuites too.

For the pytest-sentry it is necessary to specify PYTEST_SENTRY_DSN.
There are several options for implementing it in our GitHub Actions. We discussed all of them within the dev team and came to a conclusion to store the PYTEST_SENTRY_DSN as a repository secret.
For the secret to be accessible from !PR.yml it is necessary to change the environment in which this workflow runs from unprivileged to privileged (from pull_request to pull_request_target). To protect access from the forks PR to the main secrets, the gatekeeper job was introduced.
It is only possible to pass the gatekeeper for users with the "write" (or higher) permission.

In the case, the user doesn't have write access, the gatekeeper will not allow the user to perform some of the tests:

image

Related:

Refs:

@drew2a drew2a force-pushed the feature/sentry_tests branch 5 times, most recently from 84ded66 to e42d463 Compare November 21, 2022 11:09
@drew2a drew2a force-pushed the feature/sentry_tests branch 2 times, most recently from 5c33eb4 to 00fd93e Compare November 21, 2022 13:35
@drew2a drew2a force-pushed the feature/sentry_tests branch 3 times, most recently from da5e14b to a5dc706 Compare November 21, 2022 15:14
@drew2a drew2a marked this pull request as ready for review November 22, 2022 08:27
@drew2a drew2a requested review from a team, kozlovsky and xoriole and removed request for a team November 22, 2022 08:27
Copy link
Contributor

@xoriole xoriole left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@drew2a drew2a merged commit 7300366 into Tribler:main Nov 23, 2022
@drew2a drew2a deleted the feature/sentry_tests branch November 23, 2022 15:32
This was referenced Jan 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants