Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parca crashloop #352

Closed
lilic opened this issue Oct 21, 2021 · 9 comments
Closed

Parca crashloop #352

lilic opened this issue Oct 21, 2021 · 9 comments
Assignees
@kakkoyun
Labels
area/symbolization Something about symbolization bug Something isn't working

Comments

@lilic
Copy link

lilic commented Oct 21, 2021 •
edited
Loading

Hey folks, I deployed parca v0.2.0 release and get the following crashloop error:

➜ kubectl logs -n parca parca-6df694b585-cb29p
ooooooooo.
`888   `Y88.
 888   .d88'  .oooo.   oooo d8b  .ooooo.   .oooo.
 888ooo88P'  `P  )88b  `888""8P d88' `"Y8 `P  )88b
 888          .oP"888   888     888        .oP"888
 888         d8(  888   888     888   .o8 d8(  888
o888o        `Y888""8o d888b    `Y8bod8P' `Y888""8o



name=parca ts=2021-10-21T11:20:27.271462421Z caller=log.go:168 level=info msg="loading bucket configuration"
level=info name=parca ts=2021-10-21T11:20:27.271832107Z caller=server.go:85 msg="starting server" addr=:7070
level=error name=parca ts=2021-10-21T11:20:47.518687354Z caller=symbol.go:132 msg="failed to open object file" file=tmp/334830476d546a6656644163694f7a752d3662362f36474e7a6d325156336f6552756f7a58765879592f42567078344e346d56647a704361795a307637572f76636e7a51556f364e717568466554313178536c/debuginfo start=4194304 limit=22261760 offset=0 err="failed to read DWARF data: unexpected EOF"
panic: sync/atomic: store of nil value into Value

goroutine 89 [running]:
sync/atomic.(*Value).Store(0xc003180030, 0x0, 0x0)
	sync/atomic/value.go:47 +0x12c
github.com/goburrow/cache.(*entry).setValue(...)
	github.com/goburrow/cache@v0.1.4/policy.go:59
github.com/goburrow/cache.newEntry(...)
	github.com/goburrow/cache@v0.1.4/policy.go:50
github.com/goburrow/cache.(*localCache).Put(0xc0001cb600, 0x141c680, 0xc00266a060, 0x0, 0x0)
	github.com/goburrow/cache@v0.1.4/local.go:124 +0x216
github.com/parca-dev/parca/pkg/symbol.(*Symbolizer).NewLiner(0xc00007d3b0, 0xc0039aa2a0, 0xc0003a43c0, 0xb4, 0xa6, 0xc0003a43c0, 0xb4, 0x0)
	github.com/parca-dev/parca/pkg/symbol/symbol.go:111 +0x542
github.com/parca-dev/parca/pkg/debuginfo.(*Store).Symbolize(0xc00003ae00, 0x198ea08, 0xc00003ae80, 0xc0039aa2a0, 0xc003596680, 0xb, 0x10, 0x500, 0x997e65, 0x22fef20)
	github.com/parca-dev/parca/pkg/debuginfo/store.go:205 +0x376
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).symbolize(0xc0001bced0, 0x198ea08, 0xc00003ae80, 0xc00433c000, 0x8f0, 0x900, 0x0, 0x0)
	github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:87 +0x7ef
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).Run.func1(0xc000098ec0, 0xc000275eb0)
	github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:57 +0xb1
github.com/parca-dev/parca/pkg/runutil.Repeat(0x2540be400, 0xc0000f6000, 0xc002df5f28, 0x0, 0x0)
	github.com/parca-dev/parca/pkg/runutil/runutil.go:38 +0x88
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).Run(0xc0001bced0, 0x198ea08, 0xc00003ae80, 0x2540be400, 0x0, 0x0)
	github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:47 +0x8e
github.com/parca-dev/parca/pkg/parca.Run.func1(0x0, 0x0)
	github.com/parca-dev/parca/pkg/parca/parca.go:174 +0x4b
github.com/oklog/run.(*Group).Run.func1(0xc0000bd020, 0xc0000c5220, 0xc0001bcf00)
	github.com/oklog/run@v1.1.0/group.go:38 +0x27
created by github.com/oklog/run.(*Group).Run
	github.com/oklog/run@v1.1.0/group.go:37 +0xbb

This is deployed on OpenShift cluster, the previous version worked correctly.

Same error happens on my Kubernetes cluster, this is scaleaway cluster with v1.22 kube version:

kubectl logs -n parca parca-6dbfb6d457-kxfxf -p
ooooooooo.
`888   `Y88.
 888   .d88'  .oooo.   oooo d8b  .ooooo.   .oooo.
 888ooo88P'  `P  )88b  `888""8P d88' `"Y8 `P  )88b
 888          .oP"888   888     888        .oP"888
 888         d8(  888   888     888   .o8 d8(  888
o888o        `Y888""8o d888b    `Y8bod8P' `Y888""8o



name=parca ts=2021-10-21T11:27:30.541368126Z caller=log.go:168 level=info msg="loading bucket configuration"
level=info name=parca ts=2021-10-21T11:27:30.541750458Z caller=server.go:85 msg="starting server" addr=:7070
level=error name=parca ts=2021-10-21T11:28:20.712754483Z caller=symbol.go:132 msg="failed to open object file" file=tmp/59346a657852657948716658516133766d314c742f457976554561646a48664a466466787936634a422f6f47686a694a576935783850304c41794b3138552f4169636649475a584b3541684452796463667a66/debuginfo start=4194304 limit=15298560 offset=0 err="failed to read DWARF data: unexpected EOF"
panic: sync/atomic: store of nil value into Value

goroutine 92 [running]:
sync/atomic.(*Value).Store(0xc003e80030, 0x0, 0x0)
	sync/atomic/value.go:47 +0x12c
github.com/goburrow/cache.(*entry).setValue(...)
	github.com/goburrow/cache@v0.1.4/policy.go:59
github.com/goburrow/cache.newEntry(...)
	github.com/goburrow/cache@v0.1.4/policy.go:50
github.com/goburrow/cache.(*localCache).Put(0xc00029ef20, 0x141c680, 0xc0000a4110, 0x0, 0x0)
	github.com/goburrow/cache@v0.1.4/local.go:124 +0x216
github.com/parca-dev/parca/pkg/symbol.(*Symbolizer).NewLiner(0xc000296fa0, 0xc003e20ae0, 0xc003934540, 0xb4, 0xa6, 0xc003934540, 0xb4, 0x0)
	github.com/parca-dev/parca/pkg/symbol/symbol.go:111 +0x542
github.com/parca-dev/parca/pkg/debuginfo.(*Store).Symbolize(0xc0005d9e00, 0x198ea08, 0xc0005d9e80, 0xc003e20ae0, 0xc003bb9000, 0xa4, 0x100, 0x80, 0x997e65, 0x22fef20)
	github.com/parca-dev/parca/pkg/debuginfo/store.go:205 +0x376
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).symbolize(0xc00002d140, 0x198ea08, 0xc0005d9e80, 0xc003da6000, 0x355, 0x400, 0x0, 0x0)
	github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:87 +0x7ef
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).Run.func1(0xc00264cec0, 0xc00033deb0)
	github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:57 +0xb1
github.com/parca-dev/parca/pkg/runutil.Repeat(0x2540be400, 0xc0000c8000, 0xc003b99f28, 0x0, 0x0)
	github.com/parca-dev/parca/pkg/runutil/runutil.go:38 +0x88
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).Run(0xc00002d140, 0x198ea08, 0xc0005d9e80, 0x2540be400, 0x0, 0x0)
	github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:47 +0x8e
github.com/parca-dev/parca/pkg/parca.Run.func1(0x0, 0x0)
	github.com/parca-dev/parca/pkg/parca/parca.go:174 +0x4b
github.com/oklog/run.(*Group).Run.func1(0xc000029200, 0xc0004dd600, 0xc00002d170)
	github.com/oklog/run@v1.1.0/group.go:38 +0x27
created by github.com/oklog/run.(*Group).Run
	github.com/oklog/run@v1.1.0/group.go:37 +0xbb
@paulfantom paulfantom mentioned this issue Oct 21, 2021
Merged
@lilic
Copy link
Author

lilic commented Oct 22, 2021

@kakkoyun I tried the latest version but I am still running into this issue after the v0.2.1 version. It does happen later onwards:

This is a scaleway v1.22.2 cluster, its primary use is to run etcd. Let me know if you need access to it to debug or anything? Thanks!

parca-5645db7cf7-57pxv -f
ooooooooo.
`888   `Y88.
 888   .d88'  .oooo.   oooo d8b  .ooooo.   .oooo.
 888ooo88P'  `P  )88b  `888""8P d88' `"Y8 `P  )88b
 888          .oP"888   888     888        .oP"888
 888         d8(  888   888     888   .o8 d8(  888
o888o        `Y888""8o d888b    `Y8bod8P' `Y888""8o



name=parca ts=2021-10-22T07:34:30.095749681Z caller=log.go:168 level=info msg="loading bucket configuration"
level=info name=parca ts=2021-10-22T07:34:30.096186342Z caller=server.go:85 msg="starting server" addr=:7070
name=parca ts=2021-10-22T07:35:01.130597086Z caller=log.go:168 level=error protocol=grpc grpc.component=server grpc.service=parca.profilestore.v1alpha1.ProfileStoreService grpc.method=WriteRaw grpc.method_type=unary grpc.start_time=2021-10-22T07:35:01Z grpc.code=Internal grpc.error="rpc error: code = Internal desc = failed to normalize pprof: execute SQL statement: constraint failed: UNIQUE constraint failed: locations.mapping_id, locations.is_folded, locations.normalized_address, locations.lines (2067)" grpc.time_ms=43.411 finishedcall=(MISSING)
level=error name=parca ts=2021-10-22T07:35:11.375113276Z caller=symbol.go:133 msg="failed to open object file" file=tmp/56566570434952652d4c335a307862364b44416b2f7674566d48766856364153754a35695271706d612f476c343554512d71667666485f55664f693230432f3276653632464a4a422d4933564a6e506f437932/debuginfo start=4194304 limit=9695232 offset=0 err="failed to read DWARF data: unexpected EOF"
panic: runtime error: hash of unhashable type func(string) (string, error)

goroutine 74 [running]:
sync.(*Map).Load(0xc00047af78, 0x1456e20, 0x17fb9c0, 0x0, 0x0, 0x2b6a0a4401550620)
	sync/map.go:104 +0xaf
github.com/goburrow/cache.(*cache).get(0xc00047af20, 0x1456e20, 0x17fb9c0, 0xca38d8d8b9293066, 0x22a8a00)
	github.com/goburrow/cache@v0.1.4/policy.go:129 +0x5b
github.com/goburrow/cache.(*localCache).Invalidate(0xc00047af20, 0x1456e20, 0x17fb9c0)
	github.com/goburrow/cache@v0.1.4/local.go:147 +0x67
github.com/parca-dev/parca/pkg/symbol.(*Symbolizer).NewLiner(0xc00007d590, 0xc00395ede0, 0xc002e56540, 0xb4, 0xa6, 0xc002e56540, 0xb4, 0x0)
	github.com/parca-dev/parca/pkg/symbol/symbol.go:107 +0x5f6
github.com/parca-dev/parca/pkg/debuginfo.(*Store).Symbolize(0xc00003bf80, 0x198e9c8, 0xc0003e55c0, 0xc00395ede0, 0xc0003c2700, 0x11, 0x20, 0xc0121f35f0, 0x0, 0x0)
	github.com/parca-dev/parca/pkg/debuginfo/store.go:205 +0x376
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).symbolize(0xc000181da0, 0x198e9c8, 0xc0003e55c0, 0xc003954000, 0x1f4, 0x200, 0x0, 0x0)
	github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:87 +0x7ef
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).Run.func1(0xc000506ec0, 0xc000223eb0)
	github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:57 +0xb1
github.com/parca-dev/parca/pkg/runutil.Repeat(0x2540be400, 0xc00007e0c0, 0xc00b691f28, 0x0, 0x0)
	github.com/parca-dev/parca/pkg/runutil/runutil.go:38 +0x88
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).Run(0xc000181da0, 0x198e9c8, 0xc0003e55c0, 0x2540be400, 0x0, 0x0)
	github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:47 +0x8e
github.com/parca-dev/parca/pkg/parca.Run.func1(0x0, 0x0)
	github.com/parca-dev/parca/pkg/parca/parca.go:174 +0x4b
github.com/oklog/run.(*Group).Run.func1(0xc00048b5c0, 0xc000488e40, 0xc000181dd0)
	github.com/oklog/run@v1.1.0/group.go:38 +0x27
created by github.com/oklog/run.(*Group).Run
	github.com/oklog/run@v1.1.0/group.go:37 +0xbb

@kakkoyun
Copy link
Member

Just in a minute ago, I reproduced exactly the same issue. Coincidence. I'm going to send another patch version. This time I'll make sure we have tests.

Sorry for inconvenience

@lilic
Copy link
Author

lilic commented Oct 22, 2021

No problem at all! Thank you for the hard work!

@kakkoyun
Copy link
Member

kakkoyun commented Oct 22, 2021 •
edited
Loading

Hey @lilic, could you test this image? ghcr.io/parca-dev/parca:v0.2.1-1-gd4e26e9d

@metalmatze
Copy link
Member

Deployed that image just now and instead I now see:

level=error name=parca ts=2021-10-22T09:03:17.559397612Z caller=symbol.go:129 msg="failed to open object file" file=tmp/7265646163746564/debuginfo start=4194304 limit=22405120 offset=0 err="failed to read DWARF data: unexpected EOF"
panic: runtime error: hash of unhashable type func(string) (string, error)

@lilic
Copy link
Author

lilic commented Oct 22, 2021

Yes same as Matthias, here are the full logs:

level=error name=parca ts=2021-10-22T10:02:47.611773675Z caller=symbol.go:129 msg="failed to open object file" file=tmp/79687530797a49745657744241716a50715a77472f63534c6d544161377571675739655571483379532f4c55477551487a354c6157706b5f4d392d37686f2f6b323936716d494b5f6573414a662d39315f596b/debuginfo start=4194304 limit=27549696 offset=0 err="failed to read DWARF data: unexpected EOF"
panic: runtime error: hash of unhashable type func(string) (string, error)

goroutine 89 [running]:
sync.(*Map).Load(0xc000010fa0, 0x1458160, 0x17fcef0, 0x0, 0x0, 0x2b6a0a4401551b20)
	sync/map.go:104 +0xaf
github.com/goburrow/cache.(*cache).get(0xc000010f20, 0x1458160, 0x17fcef0, 0x8ed88af5245b312b, 0x22aaa00)
	github.com/goburrow/cache@v0.1.4/policy.go:129 +0x5b
github.com/goburrow/cache.(*localCache).Invalidate(0xc000010f20, 0x1458160, 0x17fcef0)
	github.com/goburrow/cache@v0.1.4/local.go:147 +0x67
github.com/parca-dev/parca/pkg/symbol.(*Symbolizer).NewLiner(0xc0002cd810, 0xc0039c04e0, 0xc00e866480, 0xb4, 0xa6, 0xc00e866480, 0xb4, 0x0)
	github.com/parca-dev/parca/pkg/symbol/symbol.go:103 +0x5f6
github.com/parca-dev/parca/pkg/debuginfo.(*Store).Symbolize(0xc00059d040, 0x1990068, 0xc00059d0c0, 0xc0039c04e0, 0xc000278e00, 0x1c, 0x20, 0xc01be654a0, 0x0, 0x0)
	github.com/parca-dev/parca/pkg/debuginfo/store.go:205 +0x376
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).symbolize(0xc0002beed0, 0x1990068, 0xc00059d0c0, 0xc00033b000, 0x12b, 0x200, 0x0, 0x0)
	github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:87 +0x7ef
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).Run.func1(0xc000353ec0, 0xc0005cdeb0)
	github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:57 +0xb1
github.com/parca-dev/parca/pkg/runutil.Repeat(0x2540be400, 0xc0000b8000, 0xc0160ddf28, 0x0, 0x0)
	github.com/parca-dev/parca/pkg/runutil/runutil.go:38 +0x88
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).Run(0xc0002beed0, 0x1990068, 0xc00059d0c0, 0x2540be400, 0x0, 0x0)
	github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:47 +0x8e
github.com/parca-dev/parca/pkg/parca.Run.func1(0x0, 0x0)
	github.com/parca-dev/parca/pkg/parca/parca.go:174 +0x4b
github.com/oklog/run.(*Group).Run.func1(0xc00009d980, 0xc000433060, 0xc0002bef00)
	github.com/oklog/run@v1.1.0/group.go:38 +0x27
created by github.com/oklog/run.(*Group).Run
	github.com/oklog/run@v1.1.0/group.go:37 +0xbb

@kakkoyun kakkoyun added the bug Something isn't working label Oct 22, 2021
@kakkoyun kakkoyun self-assigned this Oct 22, 2021
@metalmatze metalmatze mentioned this issue Oct 25, 2021
Closed
@paulfantom
Copy link
Contributor

Just updated to 0.2.2, it seems parca is not panicing anymore, but following error is still present:

level=error name=parca ts=2021-10-26T13:30:11.149890581Z caller=symbolizer.go:59 component=symbolizer msg="symbolization attempt failed" err="18 errors occurred:\n\t* storage symbolization request failed: failed to create liner: failed to initialize liner\n\t* storage

@lilic
Copy link
Author

lilic commented Oct 26, 2021

Yes, same for me as for Pawel, it does not panic but I see also different errors happen on every x interval:

level=error name=parca ts=2021-10-26T14:29:32.797954972Z caller=symbol.go:129 msg="failed to open object file" file=tmp/79687530797a49745657744241716a50715a77472f63534c6d544161377571675739655571483379532f4c55477551487a354c6157706b5f4d392d37686f2f6b323936716d494b5f6573414a662d39315f596b/debuginfo start=4194304 limit=27549696 offset=0 err="failed to read DWARF data: unexpected EOF"
level=error name=parca ts=2021-10-26T14:29:34.386539364Z caller=symbol.go:129 msg="failed to open object file" file=tmp/56566570434952652d4c335a307862364b44416b2f7674566d48766856364153754a35695271706d612f476c343554512d71667666485f55664f693230432f3276653632464a4a422d4933564a6e506f437932/debuginfo start=4194304 limit=9695232 offset=0 err="failed to read DWARF data: unexpected EOF"
level=error name=parca ts=2021-10-26T14:29:36.151237687Z caller=symbol.go:129 msg="failed to open object file" file=tmp/59346a657852657948716658516133766d314c742f457976554561646a48664a466466787936634a422f6f47686a694a576935783850304c41794b3138552f4169636649475a584b3541684452796463667a66/debuginfo start=4194304 limit=15298560 offset=0 err="failed to read DWARF data: unexpected EOF"
level=error name=parca ts=2021-10-26T14:29:36.151412959Z caller=symbolizer.go:59 component=symbolizer msg="symbolization attempt failed" err="3 errors occurred:\n\t* storage symbolization request failed: failed to create liner: failed to read DWARF data: unexpected EOF\n\t* storage symbolization request failed: failed to create liner: failed to read DWARF data: unexpected EOF\n\t* storage symbolization request failed: failed to create liner: failed to read DWARF data: unexpected EOF\n\n"
level=error name=parca ts=2021-10-26T14:29:42.754015148Z caller=symbol.go:129 msg="failed to open object file" file=tmp/4e3447534974653845644f4f2d51736a6b657a4b2f7063304478783673494c5f5a776179626c4855362f4a55654242666764716856335556794b52374d412f7a394c67664e39616a784e794a4d684b5a356e41/debuginfo start=4194304 limit=20254720 offset=0 err="failed to read DWARF data: unexpected EOF"
level=error name=parca ts=2021-10-26T14:29:42.786084933Z caller=symbolizer.go:59 component=symbolizer msg="symbolization attempt failed" err="4 errors occurred:\n\t* storage symbolization request failed: failed to create liner: failed to initialize liner\n\t* storage symbolization request failed: failed to create liner: failed to initialize liner\n\t* storage symbolization request failed: failed to create liner: failed to initialize liner\n\t* storage symbolization request failed: failed to create liner: failed to read DWARF data: unexpected EOF\n\n"
....
level=error name=parca ts=2021-10-26T14:31:32.547159971Z caller=symbolizer.go:59 component=symbolizer msg="symbolization attempt failed" err="4 errors occurred:\n\t* storage symbolization request failed: failed to create liner: failed to initialize liner\n\t* storage symbolization request failed: failed to create liner: failed to initialize liner\n\t* storage symbolization request failed: failed to create liner: failed to initialize liner\n\t* storage symbolization request failed: failed to create liner: failed to initialize liner\n\n"

@kakkoyun kakkoyun mentioned this issue Oct 27, 2021
Closed
@kakkoyun
Copy link
Member

Symbolization is best-effort right now, we're constantly working on the improvements.

That being said this is related to a bug in the Go standard library. This is on my to-do list. golang/go#18667

I'm going to close this issue for now. And I'm gonna create another issue for the errors above.

Thanks a lot for reporting.

@kakkoyun kakkoyun added the area/symbolization Something about symbolization label Feb 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kakkoyun kakkoyun

Labels
area/symbolization Something about symbolization bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@kakkoyun @metalmatze @paulfantom @lilic