Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hack/verify-all should output the specific tests that failed #3282

Closed
robscott opened this issue Aug 19, 2024 · 5 comments · Fixed by #3286
Closed

hack/verify-all should output the specific tests that failed #3282

robscott opened this issue Aug 19, 2024 · 5 comments · Fixed by #3286
Assignees
Labels
good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/feature Categorizes issue or PR as related to a new feature.

Comments

@robscott
Copy link
Member

What would you like to be added:
When a test fails in hack/verify-all.sh, we should output the failed tests at the end of the run.

Why this is needed:
Debugging test logs is very complicated. For example, consider this recent failure where the output looked like this:

�[0;32mSUCCESS: hack/../hack/verify-yamllint.sh �[0m
make: *** [Makefile:113: verify] Error 1

The actual problematic logs were:

level=warning msg="[config_reader] The configuration option `linters.govet.check-shadowing` is deprecated. Please enable `shadow` instead, if you are not using `enable-all`."
conformance/tests/backendtlspolicy-normative.go:22: File is not `goimports`-ed with -local sigs.k8s.io/gateway-api (goimports)
	"k8s.io/apimachinery/pkg/types"
�[0;31mTest FAILED: hack/../hack/verify-golint.sh �[0m

It would be great to at least show the specific script that failed closer to the final output so it's more clear why make verify failed.

@robscott robscott added the kind/feature Categorizes issue or PR as related to a new feature. label Aug 19, 2024
@robscott
Copy link
Member Author

/help
/good-first-issue

@k8s-ci-robot
Copy link
Contributor

@robscott:
This request has been marked as suitable for new contributors.

Guidelines

Please ensure that the issue body includes answers to the following questions:

  • Why are we solving this issue?
  • To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
  • Does this issue have zero to low barrier of entry?
  • How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

/help
/good-first-issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. labels Aug 19, 2024
@jgao1025
Copy link
Contributor

jgao1025 commented Aug 20, 2024

The raw file is difficult to read.

Prow has a feature that it can only list the colored line in a build.log. Something like this - https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_gateway-api/3212/pull-gateway-api-verify/1825622946182336512.

I can easily locate all errors from there, and only the problem is it has a bug. It starts to list the wrong lines on the server side after I click Analyze button even I don't have any privilege; I don't see there is any way to recover this operation.

@TheInvincibleRalph
Copy link
Contributor

Hi @robscott, I'd love to give this a shot.

@mlavacca
Copy link
Member

/assign @TheInvincibleRalph

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/feature Categorizes issue or PR as related to a new feature.
Projects
None yet
5 participants