Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Panic in canaries. #5128

Closed
cyriltovena opened this issue Jan 13, 2022 · 1 comment · Fixed by #5568
Closed

Panic in canaries. #5128

cyriltovena opened this issue Jan 13, 2022 · 1 comment · Fixed by #5568
Labels
stale A stale issue or PR that will automatically be closed. type/bug Somehing is not working as expected

Comments

@cyriltovena
Copy link
Contributor

cyriltovena commented Jan 13, 2022

While checking on a cluster I found out a canary was panicking here is stack trace:

Connecting to loki at wss://../loki/api/v1/tail?query=%7Bstream%3D%22stdout%22%2Cpod%3D%22loki-canary-7c58648dc8-xwzdr%22%7D, querying for label 'pod' with value 'loki-canary-7c58648dc8-xwzdr'
error reading websocket, will retry in 10 seconds: websocket: close 1011 (internal server error): rpc error: code = Canceled desc = grpc: the client connection is closing
Connecting to loki at wss://l...t/loki/api/v1/tail?query=%7Bstream%3D%22stdout%22%2Cpod%3D%22loki-canary-7c58648dc8-xwzdr%22%7D, querying for label 'pod' with value 'loki-canary-7c58648dc8-xwzdr'
	/src/loki/pkg/canary/comparator/comparator.go:414 +0x3a5
created by github.com/grafana/loki/pkg/canary/comparator.(*Comparator).pruneEntries
	/src/loki/pkg/canary/comparator/comparator.go:446 +0x27d
github.com/grafana/loki/pkg/canary/comparator.(*Comparator).confirmMissing(0xc000230160, {0xc00055f0e0, 0xc00055f0e0, 0x1182f00})
goroutine 1661 [running]:
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xa4209d]
panic: runtime error: invalid memory address or nil pointer dereference
@stale
Copy link

stale bot commented Mar 2, 2022

Hi! This issue has been automatically marked as stale because it has not had any
activity in the past 30 days.

We use a stalebot among other tools to help manage the state of issues in this project.
A stalebot can be very useful in closing issues in a number of cases; the most common
is closing issues or PRs where the original reporter has not responded.

Stalebots are also emotionless and cruel and can close issues which are still very relevant.

If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.

We regularly sort for closed issues which have a stale label sorted by thumbs up.

We may also:

  • Mark issues as revivable if we think it's a valid issue but isn't something we are likely
    to prioritize in the future (the issue will still remain closed).
  • Add a keepalive label to silence the stalebot if the issue is very common/popular/important.

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task,
our sincere apologies if you find yourself at the mercy of the stalebot.

@stale stale bot added the stale A stale issue or PR that will automatically be closed. label Mar 2, 2022
@chaudum chaudum added the type/bug Somehing is not working as expected label Jun 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
stale A stale issue or PR that will automatically be closed. type/bug Somehing is not working as expected
Projects
None yet
2 participants