-
Notifications
You must be signed in to change notification settings - Fork 112
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ensure failed conformance tests are caught in pipeline & fix local make command #1203
Conversation
Approved before I saw the failure..so holding off. |
|
@bjee19 We're missing failed conformance tests in the pipeline at the moment because we always return green even if there was a conformance test failure. This PR fixes that by checking if there was a failure in the test run and exiting the CI job with the exit code 2 if there is a failure. I'll update the description to be more clear 😄 |
6480f6e
to
bfe7ae9
Compare
…ke command (nginx#1203) * Fail conf tests in pipeline & fix local make command
Proposed changes
Problem: The variables in the make target for running conformance tests are not being properly evaluated in the pipeline. This means we're missing failed conformance tests in the pipeline at the moment because we always return green even if there was a conformance test failure.
Solution: This PR fixes that by checking if there was a failure in the test run and exiting the CI job with the exit code 2 if there is a failure. Also fixes the case where the target make variables are not being set correctly when the
make run-conformance-tests
are being ran locally.Testing: Tested locally and in #1169
Checklist
Before creating a PR, run through this checklist and mark each as complete.