Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix the panic error in dashboard proxy #3581

Merged
merged 1 commit into from
Sep 10, 2024

Conversation

TianxiangShen
Copy link
Contributor

@TianxiangShen TianxiangShen commented Sep 3, 2024

Changes

fix the panic error in dashboard proxy which is due to null pointer. The external log server might returns an empty response. We need to check the response first before returns the status code.

{"level":"error","ts":1724741273.0200272,"caller":"utils/proxy.go:46","msg":"Failed to execute request: Get \"http://pod-logs-server:80/tnet-cd/d89f4a51-6439-11ef-bac3-7ee28d663481-pr-checkout-pod/step-pr-checkout?startTime=2024-08-27T06%3A01%3A48Z&completionTime=2024-08-27T06%3A02%3A22Z\": context canceled","stacktrace":"github.com/tektoncd/dashboard/pkg/utils.Proxy\n\tgithub.com/tektoncd/dashboard/pkg/utils/proxy.go:46\ngithub.com/tektoncd/dashboard/pkg/endpoints.Resource.LogsProxy\n\tgithub.com/tektoncd/dashboard/pkg/endpoints/externallogs.go:34\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2109\nnet/http.(*ServeMux).ServeHTTP\n\tnet/http/server.go:2487\ngithub.com/tektoncd/dashboard/pkg/csrf.(*csrf).ServeHTTP\n\tgithub.com/tektoncd/dashboard/pkg/csrf/csrf.go:74\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2947\nnet/http.(*conn).serve\n\tnet/http/server.go:1991"}
2024/08/27 06:47:53 http: panic serving 10.222.16.105:29181: runtime error: invalid memory address or nil pointer dereference
goroutine 82193969 [running]:
net/http.(*conn).serve.func1()
	net/http/server.go:1850 +0xbf
panic({0x15f31e0, 0x252e9f0})
	runtime/panic.go:890 +0x262
github.com/tektoncd/dashboard/pkg/utils.Proxy(0xc002709a00, {0x1a23098, 0xc000a26e00}, {0xc0022069c0, 0xb2}, 0xc00220695d?)
	github.com/tektoncd/dashboard/pkg/utils/proxy.go:47 +0x537
github.com/tektoncd/dashboard/pkg/endpoints.Resource.LogsProxy({0xc0004f4240, {0x1a35fb8, 0xc00058a340}, {{0xc000046074, 0x10}, {0x7ffd15ea4508, 0x10}, {0x7ffd15ea452e, 0x10}, {0x7ffd15ea4575, ...}, ...}}, ...)
	github.com/tektoncd/dashboard/pkg/endpoints/externallogs.go:34 +0x13d
net/http.HandlerFunc.ServeHTTP(0x4a?, {0x1a23098?, 0xc000a26e00?}, 0x0?)
	net/http/server.go:2109 +0x2f
net/http.(*ServeMux).ServeHTTP(0x15d8600?, {0x1a23098, 0xc000a26e00}, 0xc002709a00)
	net/http/server.go:2487 +0x149
github.com/tektoncd/dashboard/pkg/csrf.(*csrf).ServeHTTP(0xc000379410, {0x1a23098, 0xc000a26e00}, 0xc002709a00)
	github.com/tektoncd/dashboard/pkg/csrf/csrf.go:74 +0x94
net/http.serverHandler.ServeHTTP({0xc001f38450?}, {0x1a23098, 0xc000a26e00}, 0xc002709a00)
	net/http/server.go:2947 +0x30c
net/http.(*conn).serve(0xc0010a77c0, {0x1a23ba0, 0xc000379500})
	net/http/server.go:1991 +0x607
created by net/http.(*Server).Serve
	net/http/server.go:3102 +0x4db

Submitter Checklist

As the author of this PR, please check off the items in this checklist:

  • Docs included if any changes are user facing
  • Tests included if any functionality added or changed
  • Follows the commit message standard
  • Meets the Tekton contributor standards (including
    functionality, content, code)
  • Has a kind label. You can add one by adding a comment on this PR that contains /kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tep
  • Release notes block below has been updated with any user facing changes (new features, significant UI changes, API changes, bug fixes, changes requiring upgrade notices or deprecation warnings)
  • Release notes contains the string "action required" if the change requires additional action from users switching to the new release

Release Notes

Fix the panic error in dashboard proxy

Copy link

linux-foundation-easycla bot commented Sep 3, 2024

CLA Signed

The committers listed above are authorized under a signed CLA.

  • ✅ login: TianxiangShen / name: Tianxiang Shen (b026fce)

@tekton-robot tekton-robot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Sep 3, 2024
@tekton-robot
Copy link
Contributor

Hi @TianxiangShen. Thanks for your PR.

I'm waiting for a tektoncd member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@tekton-robot tekton-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Sep 3, 2024
@TianxiangShen
Copy link
Contributor Author

/kind bug

@tekton-robot tekton-robot added the kind/bug Categorizes issue or PR as related to a bug. label Sep 3, 2024
@AlanGreene
Copy link
Member

Thanks for the PR @TianxiangShen. Can you share some details about how to reproduce the issue?

We'll be delivering a complete rewrite of the backend in an upcoming release so it would be good to verify the new version is not affected by the same issue.

/ok-to-test

@tekton-robot tekton-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Sep 4, 2024
@TianxiangShen
Copy link
Contributor Author

Thanks for the PR @TianxiangShen. Can you share some details about how to reproduce the issue?

We'll be delivering a complete rewrite of the backend in an upcoming release so it would be good to verify the new version is not affected by the same issue.

/ok-to-test

Hi Alan, sorry for the late reply. This issue happens when dashboard backend trys to access the external-logs server and that server send backs an empty respons. here's my setting for the dashboard pod:

spec:
  containers:
  - args:
    - --port=9097
    - --logout-url=
    - --pipelines-namespace=tekton-pipelines
    - --triggers-namespace=tekton-pipelines
    - --read-only=true
    - --log-level=info
    - --log-format=json
    - --namespace=
    - --stream-logs=true
    - --external-logs=http://pod-logs-server:80
...

when the external-logs server http://pod-logs-server:80 failed to fetch the logs from our storage bucket, it shows following error and the response will be empty.

2024/08/16 02:18:03 failed to check existence of py-cd/abc-pod/step-checkout: failed to stat object: Head "https://bucket.nuobject.io/tekton-logs/py-cd/abc-pod/step-checkout": context deadline exceeded

@AlanGreene
Copy link
Member

Thanks for the extra info @TianxiangShen, this is good to know.

/approve
/lgtm

@tekton-robot tekton-robot added the lgtm Indicates that a PR is ready to be merged. label Sep 10, 2024
@tekton-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: AlanGreene

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@tekton-robot tekton-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 10, 2024
@tekton-robot tekton-robot merged commit 7ac4ba5 into tektoncd:main Sep 10, 2024
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. kind/bug Categorizes issue or PR as related to a bug. lgtm Indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants