Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

receive: reduce locking #114

Merged
merged 4 commits into from
Aug 28, 2024
Merged

receive: reduce locking #114

merged 4 commits into from
Aug 28, 2024

Conversation

GiedriusS
Copy link
Collaborator

@GiedriusS GiedriusS commented Aug 27, 2024 •
edited
Loading

blocksToDelete() should only be called when the TSDB is running so
there's no point in locking and checking here. Also, removed shipper()
as it is buggy. We want to hold down the lock until pruning is done.

Trimmed goroutine dump:

goroutine profile: total 11252
5148 @ 0x43e48e 0x44f938 0x44f90f 0x46d685 0x1ef18aa 0x1ef1885 0x1ef55d3 0x1c7700e 0x1f8dd37 0x1e57b22 0x16a35d2 0x1450450 0x1e5deb7 0x15f4410 0x161f6bb 0x1e5deb7 0x144e850 0x1e5deb7 0x1e5df83 0x1e5deb7 0x1e586fc 0x1e5deb7 0x1e5dd55 0x16a34b6 0xf17c55 0xf1bbe7 0xf15d2d 0x4716c1
#	0x46d684	sync.runtime_SemacquireRWMutexR+0x24												/usr/lib/golang/src/runtime/sema.go:82
#	0x1ef18a9	sync.(*RWMutex).RLock+0x49													/usr/lib/golang/src/sync/rwmutex.go:71
#	0x1ef1884	github.com/thanos-io/thanos/pkg/receive.(*tenant).client+0x24									/home/giedriusstatkevicius/dev/thanos/pkg/receive/multitsdb.go:281
#	0x1ef55d2	github.com/thanos-io/thanos/pkg/receive.(*MultiTSDB).TSDBLocalClients+0x172							/home/giedriusstatkevicius/dev/thanos/pkg/receive/multitsdb.go:620
#	0x1c7700d	github.com/thanos-io/thanos/pkg/store.(*ProxyStore).LabelSet+0x2d								/home/giedriusstatkevicius/dev/thanos/pkg/store/proxy.go:219
#	0x1f8dd36	main.runReceive.func3+0x16													/home/giedriusstatkevicius/dev/thanos/cmd/thanos/receive.go:344
#	0x1e57b21	github.com/thanos-io/thanos/pkg/info.(*InfoServer).Info+0x21									/home/giedriusstatkevicius/dev/thanos/pkg/info/info.go:174
#	0x16a35d1	github.com/thanos-io/thanos/pkg/info/infopb._Info_Info_Handler.func1+0x71							/home/giedriusstatkevicius/dev/thanos/pkg/info/infopb/rpc.pb.go:507
#	0x145044f	github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/logging.UnaryServerInterceptor.UnaryServerInterceptor.func1+0x20f	/home/giedriusstatkevicius/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/server.go:22
#	0x1e5deb6	github.com/thanos-io/thanos/pkg/server/grpc.New.WithUnaryServerChain.ChainUnaryServer.func10.1.1+0x36				/home/giedriusstatkevicius/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:27
#	0x15f440f	github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/tracing.UnaryServerInterceptor.UnaryServerInterceptor.func1+0x20f	/home/giedriusstatkevicius/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/server.go:22
#	0x161f6ba	github.com/thanos-io/thanos/pkg/tracing.UnaryServerInterceptor.func1+0x7a							/home/giedriusstatkevicius/dev/thanos/pkg/tracing/grpc.go:30
#	0x1e5deb6	github.com/thanos-io/thanos/pkg/server/grpc.New.WithUnaryServerChain.ChainUnaryServer.func10.1.1+0x36				/home/giedriusstatkevicius/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:27
#	0x144e84f	github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/tags.UnaryServerInterceptor.UnaryServerInterceptor.func1+0x20f	/home/giedriusstatkevicius/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/server.go:22
#	0x1e5deb6	github.com/thanos-io/thanos/pkg/server/grpc.New.WithUnaryServerChain.ChainUnaryServer.func10.1.1+0x36				/home/giedriusstatkevicius/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:27
#	0x1e5df82	github.com/thanos-io/thanos/pkg/server/grpc.New.(*ServerMetrics).UnaryServerInterceptor.func6+0x82				/home/giedriusstatkevicius/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:107
#	0x1e5deb6	github.com/thanos-io/thanos/pkg/server/grpc.New.WithUnaryServerChain.ChainUnaryServer.func10.1.1+0x36				/home/giedriusstatkevicius/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:27
#	0x1e586fb	github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/recovery.UnaryServerInterceptor.func1+0x9b				/home/giedriusstatkevicius/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/recovery/interceptors.go:31
#	0x1e5deb6	github.com/thanos-io/thanos/pkg/server/grpc.New.WithUnaryServerChain.ChainUnaryServer.func10.1.1+0x36				/home/giedriusstatkevicius/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:27
#	0x1e5dd54	github.com/thanos-io/thanos/pkg/server/grpc.New.WithUnaryServerChain.ChainUnaryServer.func10+0xb4				/home/giedriusstatkevicius/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:36
#	0x16a34b5	github.com/thanos-io/thanos/pkg/info/infopb._Info_Info_Handler+0x115								/home/giedriusstatkevicius/dev/thanos/pkg/info/infopb/rpc.pb.go:509
#	0xf17c54	google.golang.org/grpc.(*Server).processUnaryRPC+0xcd4										/home/giedriusstatkevicius/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:1282
#	0xf1bbe6	google.golang.org/grpc.(*Server).handleStream+0x9e6										/home/giedriusstatkevicius/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:1619
#	0xf15d2c	google.golang.org/grpc.(*Server).serveStreams.func1.2+0x8c									/home/giedriusstatkevicius/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:921

3320 @ 0x43e48e 0x44f938 0x44f90f 0x46d685 0x1ef18aa 0x1ef1885 0x1ef55d3 0x1c77c09 0x1c82662 0x1c6bab1 0x1c81258 0x15cb690 0x145091a 0x1e5d977 0x15f48da 0x161f97a 0x1e5d977 0x144ed3a 0x1e5d977 0x1e5da92 0x1e5d977 0x1e589d3 0x1e5d977 0x1e5d815 0xf1a471 0xf1bba6 0xf15d2d 0x4716c1
#	0x46d684	sync.runtime_SemacquireRWMutexR+0x24													/usr/lib/golang/src/runtime/sema.go:82
#	0x1ef18a9	sync.(*RWMutex).RLock+0x49														/usr/lib/golang/src/sync/rwmutex.go:71
#	0x1ef1884	github.com/thanos-io/thanos/pkg/receive.(*tenant).client+0x24										/home/giedriusstatkevicius/dev/thanos/pkg/receive/multitsdb.go:281
#	0x1ef55d2	github.com/thanos-io/thanos/pkg/receive.(*MultiTSDB).TSDBLocalClients+0x172								/home/giedriusstatkevicius/dev/thanos/pkg/receive/multitsdb.go:620
#	0x1c77c08	github.com/thanos-io/thanos/pkg/store.(*ProxyStore).Series+0x568									/home/giedriusstatkevicius/dev/thanos/pkg/store/proxy.go:323
#	0x1c82661	github.com/thanos-io/thanos/pkg/store.(*instrumentedStoreServer).Series+0xa1								/home/giedriusstatkevicius/dev/thanos/pkg/store/telemetry.go:169
#	0x1c6bab0	github.com/thanos-io/thanos/pkg/store.(*limitedStoreServer).Series+0x1b0								/home/giedriusstatkevicius/dev/thanos/pkg/store/limiter.go:141
#	0x1c81257	github.com/thanos-io/thanos/pkg/store.(*recoverableStoreServer).Series+0x77								/home/giedriusstatkevicius/dev/thanos/pkg/store/recover.go:28
#	0x15cb68f	github.com/thanos-io/thanos/pkg/store/storepb._Store_Series_Handler+0xcf								/home/giedriusstatkevicius/dev/thanos/pkg/store/storepb/rpc.pb.go:1150
#	0x1450919	github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/logging.StreamServerInterceptor.StreamServerInterceptor.func1+0x279	/home/giedriusstatkevicius/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/server.go:35
#	0x1e5d976	github.com/thanos-io/thanos/pkg/server/grpc.New.WithStreamServerChain.ChainStreamServer.func12.1.1+0x36					/home/giedriusstatkevicius/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:51
#	0x15f48d9	github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/tracing.StreamServerInterceptor.StreamServerInterceptor.func1+0x279	/home/giedriusstatkevicius/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/server.go:35
#	0x161f979	github.com/thanos-io/thanos/pkg/tracing.StreamServerInterceptor.func1+0xf9								/home/giedriusstatkevicius/dev/thanos/pkg/tracing/grpc.go:42
#	0x1e5d976	github.com/thanos-io/thanos/pkg/server/grpc.New.WithStreamServerChain.ChainStreamServer.func12.1.1+0x36					/home/giedriusstatkevicius/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:51
#	0x144ed39	github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/tags.StreamServerInterceptor.StreamServerInterceptor.func1+0x279		/home/giedriusstatkevicius/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/server.go:35
#	0x1e5d976	github.com/thanos-io/thanos/pkg/server/grpc.New.WithStreamServerChain.ChainStreamServer.func12.1.1+0x36					/home/giedriusstatkevicius/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:51
#	0x1e5da91	github.com/thanos-io/thanos/pkg/server/grpc.New.(*ServerMetrics).StreamServerInterceptor.func8+0xd1					/home/giedriusstatkevicius/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:121
#	0x1e5d976	github.com/thanos-io/thanos/pkg/server/grpc.New.WithStreamServerChain.ChainStreamServer.func12.1.1+0x36					/home/giedriusstatkevicius/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:51
#	0x1e589d2	github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/recovery.StreamServerInterceptor.func1+0x92				/home/giedriusstatkevicius/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/recovery/interceptors.go:45
#	0x1e5d976	github.com/thanos-io/thanos/pkg/server/grpc.New.WithStreamServerChain.ChainStreamServer.func12.1.1+0x36					/home/giedriusstatkevicius/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:51
#	0x1e5d814	github.com/thanos-io/thanos/pkg/server/grpc.New.WithStreamServerChain.ChainStreamServer.func12+0xb4					/home/giedriusstatkevicius/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:60
#	0xf1a470	google.golang.org/grpc.(*Server).processStreamingRPC+0xeb0										/home/giedriusstatkevicius/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:1548
#	0xf1bba5	google.golang.org/grpc.(*Server).handleStream+0x9a5											/home/giedriusstatkevicius/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:1623
#	0xf15d2c	google.golang.org/grpc.(*Server).serveStreams.func1.2+0x8c										/home/giedriusstatkevicius/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:921


30 @ 0x43e48e 0x44e905 0x1c82c1d 0x4716c1
#	0x1c82c1c	github.com/thanos-io/thanos/pkg/store.NewTSDBStore.func3+0x9c	/home/giedriusstatkevicius/dev/thanos/pkg/store/tsdb.go:114

30 @ 0x43e48e 0x44e905 0xcef4d9 0x4716c1
#	0xcef4d8	github.com/prometheus/prometheus/tsdb.(*DB).run+0x178	/home/giedriusstatkevicius/go/pkg/mod/github.com/vinted/prometheus@v1.8.2-0.20240419110424-62c5d536d12f/tsdb/db.go:990


Waiting for prune to finish:
1 @ 0x43e48e 0x44f938 0x44f90f 0x46d565 0x47e3a8 0x1ef337f 0x1f8cf15 0x1623b82 0x1f8cecc 0x623309 0x4716c1
#	0x46d564	sync.runtime_Semacquire+0x24						/usr/lib/golang/src/runtime/sema.go:62
#	0x47e3a7	sync.(*WaitGroup).Wait+0x47						/usr/lib/golang/src/sync/waitgroup.go:116
#	0x1ef337e	github.com/thanos-io/thanos/pkg/receive.(*MultiTSDB).Prune+0x31e	/home/giedriusstatkevicius/dev/thanos/pkg/receive/multitsdb.go:442
#	0x1f8cf14	main.runReceive.func11.1+0x34						/home/giedriusstatkevicius/dev/thanos/cmd/thanos/receive.go:421
#	0x1623b81	github.com/thanos-io/thanos/pkg/runutil.Repeat+0x81			/home/giedriusstatkevicius/dev/thanos/pkg/runutil/runutil.go:74
#	0x1f8cecb	main.runReceive.func11+0xab						/home/giedriusstatkevicius/dev/thanos/cmd/thanos/receive.go:420
#	0x623308	github.com/oklog/run.(*Group).Run.func1+0x28				/home/giedriusstatkevicius/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38

1 @ 0x43e48e 0x44f938 0x44f90f 0x46d685 0x1ef14b7 0x1ef1493 0xcf2a1a 0xcef53d 0x4716c1
#	0x46d684	sync.runtime_SemacquireRWMutexR+0x24					/usr/lib/golang/src/runtime/sema.go:82
#	0x1ef14b6	sync.(*RWMutex).RLock+0x76						/usr/lib/golang/src/sync/rwmutex.go:71
#	0x1ef1492	github.com/thanos-io/thanos/pkg/receive.(*tenant).blocksToDelete+0x52	/home/giedriusstatkevicius/dev/thanos/pkg/receive/multitsdb.go:241
#	0xcf2a19	github.com/prometheus/prometheus/tsdb.(*DB).reloadBlocks+0x159		/home/giedriusstatkevicius/go/pkg/mod/github.com/vinted/prometheus@v1.8.2-0.20240419110424-62c5d536d12f/tsdb/db.go:1441
#	0xcef53c	github.com/prometheus/prometheus/tsdb.(*DB).run+0x1dc			/home/giedriusstatkevicius/go/pkg/mod/github.com/vinted/prometheus@v1.8.2-0.20240419110424-62c5d536d12f/tsdb/db.go:993

1 @ 0x43e48e 0x44f938 0x44f90f 0x46d685 0x1ef1b6d 0x1ef1b48 0x1ef4a05 0x1f90905 0x1f900e5 0x623309 0x4716c1
#	0x46d684	sync.runtime_SemacquireRWMutexR+0x24				/usr/lib/golang/src/runtime/sema.go:82
#	0x1ef1b6c	sync.(*RWMutex).RLock+0x4c					/usr/lib/golang/src/sync/rwmutex.go:71
#	0x1ef1b47	github.com/thanos-io/thanos/pkg/receive.(*tenant).shipper+0x27	/home/giedriusstatkevicius/dev/thanos/pkg/receive/multitsdb.go:299
#	0x1ef4a04	github.com/thanos-io/thanos/pkg/receive.(*MultiTSDB).Sync+0x364	/home/giedriusstatkevicius/dev/thanos/pkg/receive/multitsdb.go:568
#	0x1f90904	main.startTSDBAndUpload.func3+0x144				/home/giedriusstatkevicius/dev/thanos/cmd/thanos/receive.go:666
#	0x1f900e4	main.startTSDBAndUpload.func4+0x3e4				/home/giedriusstatkevicius/dev/thanos/cmd/thanos/receive.go:721
#	0x623308	github.com/oklog/run.(*Group).Run.func1+0x28			/home/giedriusstatkevicius/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38

1 @ 0x43e48e 0x44f938 0x44f90f 0x46d685 0xcef1a5 0xcef179 0x1c834fc 0x1ef09c7 0x1ef18ea 0x1ef55d3 0x1c77c09 0x1c82662 0x1c6bab1 0x1c81258 0x15cb690 0x145091a 0x1e5d977 0x15f48da 0x161f97a 0x1e5d977 0x144ed3a 0x1e5d977 0x1e5da92 0x1e5d977 0x1e589d3 0x1e5d977 0x1e5d815 0xf1a471 0xf1bba6 0xf15d2d 0x4716c1
#	0x46d684	sync.runtime_SemacquireRWMutexR+0x24													/usr/lib/golang/src/runtime/sema.go:82
#	0xcef1a4	sync.(*RWMutex).RLock+0x64														/usr/lib/golang/src/sync/rwmutex.go:71
#	0xcef178	github.com/prometheus/prometheus/tsdb.(*DB).StartTime+0x38										/home/giedriusstatkevicius/go/pkg/mod/github.com/vinted/prometheus@v1.8.2-0.20240419110424-62c5d536d12f/tsdb/db.go:964
#	0x1c834fb	github.com/thanos-io/thanos/pkg/store.(*TSDBStore).TimeRange+0x1b									/home/giedriusstatkevicius/dev/thanos/pkg/store/tsdb.go:197
#	0x1ef09c6	github.com/thanos-io/thanos/pkg/receive.newLocalClient+0x26										/home/giedriusstatkevicius/dev/thanos/pkg/receive/multitsdb.go:111
#	0x1ef18e9	github.com/thanos-io/thanos/pkg/receive.(*tenant).client+0x89										/home/giedriusstatkevicius/dev/thanos/pkg/receive/multitsdb.go:289
#	0x1ef55d2	github.com/thanos-io/thanos/pkg/receive.(*MultiTSDB).TSDBLocalClients+0x172								/home/giedriusstatkevicius/dev/thanos/pkg/receive/multitsdb.go:620
#	0x1c77c08	github.com/thanos-io/thanos/pkg/store.(*ProxyStore).Series+0x568									/home/giedriusstatkevicius/dev/thanos/pkg/store/proxy.go:323
#	0x1c82661	github.com/thanos-io/thanos/pkg/store.(*instrumentedStoreServer).Series+0xa1								/home/giedriusstatkevicius/dev/thanos/pkg/store/telemetry.go:169
#	0x1c6bab0	github.com/thanos-io/thanos/pkg/store.(*limitedStoreServer).Series+0x1b0								/home/giedriusstatkevicius/dev/thanos/pkg/store/limiter.go:141
#	0x1c81257	github.com/thanos-io/thanos/pkg/store.(*recoverableStoreServer).Series+0x77								/home/giedriusstatkevicius/dev/thanos/pkg/store/recover.go:28
#	0x15cb68f	github.com/thanos-io/thanos/pkg/store/storepb._Store_Series_Handler+0xcf								/home/giedriusstatkevicius/dev/thanos/pkg/store/storepb/rpc.pb.go:1150
#	0x1450919	github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/logging.StreamServerInterceptor.StreamServerInterceptor.func1+0x279	/home/giedriusstatkevicius/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/server.go:35
#	0x1e5d976	github.com/thanos-io/thanos/pkg/server/grpc.New.WithStreamServerChain.ChainStreamServer.func12.1.1+0x36					/home/giedriusstatkevicius/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:51
#	0x15f48d9	github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/tracing.StreamServerInterceptor.StreamServerInterceptor.func1+0x279	/home/giedriusstatkevicius/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/server.go:35
#	0x161f979	github.com/thanos-io/thanos/pkg/tracing.StreamServerInterceptor.func1+0xf9								/home/giedriusstatkevicius/dev/thanos/pkg/tracing/grpc.go:42
#	0x1e5d976	github.com/thanos-io/thanos/pkg/server/grpc.New.WithStreamServerChain.ChainStreamServer.func12.1.1+0x36					/home/giedriusstatkevicius/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:51
#	0x144ed39	github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/tags.StreamServerInterceptor.StreamServerInterceptor.func1+0x279		/home/giedriusstatkevicius/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/server.go:35
#	0x1e5d976	github.com/thanos-io/thanos/pkg/server/grpc.New.WithStreamServerChain.ChainStreamServer.func12.1.1+0x36					/home/giedriusstatkevicius/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:51
#	0x1e5da91	github.com/thanos-io/thanos/pkg/server/grpc.New.(*ServerMetrics).StreamServerInterceptor.func8+0xd1					/home/giedriusstatkevicius/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:121
#	0x1e5d976	github.com/thanos-io/thanos/pkg/server/grpc.New.WithStreamServerChain.ChainStreamServer.func12.1.1+0x36					/home/giedriusstatkevicius/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:51
#	0x1e589d2	github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/recovery.StreamServerInterceptor.func1+0x92				/home/giedriusstatkevicius/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/recovery/interceptors.go:45
#	0x1e5d976	github.com/thanos-io/thanos/pkg/server/grpc.New.WithStreamServerChain.ChainStreamServer.func12.1.1+0x36					/home/giedriusstatkevicius/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:51
#	0x1e5d814	github.com/thanos-io/thanos/pkg/server/grpc.New.WithStreamServerChain.ChainStreamServer.func12+0xb4					/home/giedriusstatkevicius/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:60
#	0xf1a470	google.golang.org/grpc.(*Server).processStreamingRPC+0xeb0										/home/giedriusstatkevicius/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:1548
#	0xf1bba5	google.golang.org/grpc.(*Server).handleStream+0x9a5											/home/giedriusstatkevicius/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:1623
#	0xf15d2c	google.golang.org/grpc.(*Server).serveStreams.func1.2+0x8c										/home/giedriusstatkevicius/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:921

1 @ 0x43e48e 0x44f938 0x44f90f 0x46d6e5 0x47dfaa 0x1ef3e1e 0x1ef385d 0x4716c1
#	0x46d6e4	sync.runtime_SemacquireRWMutex+0x24					/usr/lib/golang/src/runtime/sema.go:87
#	0x47dfa9	sync.(*RWMutex).Lock+0x69						/usr/lib/golang/src/sync/rwmutex.go:152
#	0x1ef3e1d	github.com/thanos-io/thanos/pkg/receive.(*MultiTSDB).pruneTSDB+0x33d	/home/giedriusstatkevicius/dev/thanos/pkg/receive/multitsdb.go:493
#	0x1ef385c	github.com/thanos-io/thanos/pkg/receive.(*MultiTSDB).Prune.func1+0x13c	/home/giedriusstatkevicius/dev/thanos/pkg/receive/multitsdb.go:429

@GiedriusS GiedriusS force-pushed the fix_lock branch 8 times, most recently from e070360 to 0cbe697 Compare August 28, 2024 08:21
@GiedriusS GiedriusS changed the title receive: fix taking lock receive/multitsdb: don't forget to close the store Aug 28, 2024
@GiedriusS GiedriusS force-pushed the fix_lock branch 4 times, most recently from 7542f2d to 4bfedae Compare August 28, 2024 09:58
@GiedriusS
receive: reduce locking
7d8d114 
blocksToDelete() should only be called when the TSDB is running so
there's no point in locking and checking here.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
@GiedriusS GiedriusS changed the title receive/multitsdb: don't forget to close the store receive: reduce locking Aug 28, 2024
GiedriusS and others added 3 commits August 28, 2024 15:10
@GiedriusS
receive/multitsdb: remove shipper method
b90ece5 
Remove shipper() method because we want to hold down the lock while
shipper is doing its stuff. This is because Sync() is not concurrency
safe and pruning might happen in the background.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
@GiedriusS
receive: fix linter
e95fa4c 
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
@harry671003 @GiedriusS
Fix failing e2e test (thanos-io#7620)
f1608c0 
Signed-off-by: 🌲 Harry 🌊 John 🏔 <johrry@amazon.com>
@GiedriusS GiedriusS marked this pull request as ready for review August 28, 2024 12:31
@GiedriusS GiedriusS merged commit 1699519 into 0.34.0+vinted Aug 28, 2024
12 checks passed
@GiedriusS GiedriusS deleted the fix_lock branch August 28, 2024 12:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@niaurys niaurys niaurys approved these changes

@ignasr ignasr ignasr approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@GiedriusS @ignasr @niaurys @harry671003