Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

QGB investigate concurrent map reads panic #518

Closed
Tracked by #301
rach-id opened this issue Jul 4, 2022 · 1 comment
Closed
Tracked by #301

QGB investigate concurrent map reads panic #518

rach-id opened this issue Jul 4, 2022 · 1 comment

Comments

@rach-id
Copy link
Member

rach-id commented Jul 4, 2022

Currently, sometimes when starting the QGB, we have the following panic:

core0         | 12:30AM INF commit synced commit=436F6D6D697449447B5B3139352036372036302032333520393020313934203139203233382031343120353220313839203736203833203135382031393020323237203235332036312039332031353120323136203139382031303920323534203136342038312031313820313838203531203636203132312038305D3A3233427D
core0         | 12:30AM INF committed state app_hash=C3433CEB5AC213EE8D34BD4C539EBEE3FD3D5D97D8C66DFEA45176BC33427950 height=571 module=state num_txs=4
core0         | 12:30AM INF indexed block height=571 module=txindex
core0         | 12:30AM INF Timed out dur=2976.086994 height=572 module=consensus round=0 step=1
core0         | 12:30AM INF received proposal module=consensus proposal={"Type":32,"block_id":{"hash":"7F536DCF9F7F0DCC3C9F14AB065E7D661E4EA1CC2ADBB71B5F62471AEBB07094","parts":{"hash":"1D5154803C8F493BF40648854590B08FBFF799814E891CCF3A9414CB2C2337AF","total":1}},"height":572,"pol_round":-1,"round":0,"signature":"+iTi0GZtmmzkqx/Vh3koUXSCX6gHqw+mhhvFm2gFK2975lPWDjSsnlXfAwhnzRPAAsW7+UDnbamOVd6aRujIBQ==","timestamp":"2022-06-23T00:30:25.364996368Z"}
core0         | 12:30AM INF received complete proposal block hash=7F536DCF9F7F0DCC3C9F14AB065E7D661E4EA1CC2ADBB71B5F62471AEBB07094 height=572 module=consensus
core0         | 12:30AM INF finalizing commit of block hash=7F536DCF9F7F0DCC3C9F14AB065E7D661E4EA1CC2ADBB71B5F62471AEBB07094 height=572 module=consensus num_txs=0 root=C3433CEB5AC213EE8D34BD4C539EBEE3FD3D5D97D8C66DFEA45176BC33427950
core0         | 12:30AM INF minted coins from module account amount=82397464uceles from=mint module=x/bank
core0         | 12:30AM INF executed block height=572 module=state num_invalid_txs=0 num_valid_txs=0
core0         | 12:30AM INF commit synced commit=436F6D6D697449447B5B3634203130392031393720313020313037203131352032313820313120342031343220313435203439203935203133312031333820333220313034203234302031362031313120313939203138372039392035392031313220313837203130322039372032333220363720313239203134395D3A3233437D
core0         | 12:30AM INF committed state app_hash=406DC50A6B73DA0B048E91315F838A2068F0106FC7BB633B70BB6661E8438195 height=572 module=state num_txs=0
core0         | 12:30AM INF indexed block height=572 module=txindex
core0         | 12:30AM INF Timed out dur=2988.505433 height=573 module=consensus round=0 step=1
core0         | 12:30AM INF received proposal module=consensus proposal={"Type":32,"block_id":{"hash":"6CCE5EEDA9146FCB7FB34A9F1338231000532FA1B7F1BA914CE1C0F36FB40857","parts":{"hash":"43363B3149A5A719DFD82DC3CFC4CB1ADAA03FA2306BA499299E9014F646EF28","total":1}},"height":573,"pol_round":-1,"round":0,"signature":"8nqNKQW6YcF+U3pjehGNLvjLtz1KDfBM4aS5pq9WlOFKR8X06cvutj5KscjXxWnx9Xiwhy+ct25F8JCgFzA7Ag==","timestamp":"2022-06-23T00:30:28.779696372Z"}
core0         | 12:30AM INF received complete proposal block hash=6CCE5EEDA9146FCB7FB34A9F1338231000532FA1B7F1BA914CE1C0F36FB40857 height=573 module=consensus
core0         | 12:30AM INF finalizing commit of block hash=6CCE5EEDA9146FCB7FB34A9F1338231000532FA1B7F1BA914CE1C0F36FB40857 height=573 module=consensus num_txs=0 root=406DC50A6B73DA0B048E91315F838A2068F0106FC7BB633B70BB6661E8438195
core0         | 12:30AM INF minted coins from module account amount=82397478uceles from=mint module=x/bank
core0         | 12:30AM INF executed block height=573 module=state num_invalid_txs=0 num_valid_txs=0
core0         | fatal error: concurrent map read and map write
core0         | 
core0         | goroutine 87278 [running]:
core0         | runtime.throw({0x1bd8082, 0x7f43efa47a68})
core0         | 	/usr/local/go/src/runtime/panic.go:1198 +0x71 fp=0xc0057742d8 sp=0xc0057742a8 pc=0x438ed1
core0         | runtime.mapaccess2_fast64(0xc0001f4220, 0x1995c00, 0x23c)
core0         | 	/usr/local/go/src/runtime/map_fast64.go:61 +0x176 fp=0xc0057742f8 sp=0xc0057742d8 pc=0x414c76
core0         | github.com/cosmos/iavl.(*MutableTree).VersionExists(0xc000578a00, 0x203de90)
core0         | 	/go/pkg/mod/github.com/cosmos/iavl@v0.17.1/mutable_tree.go:72 +0xac fp=0xc005774348 sp=0xc0057742f8 pc=0xcf19cc
core0         | github.com/cosmos/cosmos-sdk/store/iavl.(*Store).VersionExists(...)
core0         | 	/go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.0.1-sdk-v0.44.2.0.20220509210332-5630422e5cdf/store/iavl/store.go:145
core0         | github.com/cosmos/cosmos-sdk/store/iavl.(*Store).GetImmutable(0xc00100b790, 0x203de90)
core0         | 	/go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.0.1-sdk-v0.44.2.0.20220509210332-5630422e5cdf/store/iavl/store.go:93 +0x35 fp=0xc005774380 sp=0xc005774348 pc=0xd0c895
core0         | github.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).CacheMultiStoreWithVersion(0xc008b40640, 0xc000eca6b0)
core0         | 	/go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.0.1-sdk-v0.44.2.0.20220509210332-5630422e5cdf/store/rootmulti/store.go:473 +0x21c fp=0xc0057746b8 sp=0xc005774380 pc=0xd1785c
core0         | github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).createQueryContext(_, _, _)
core0         | 	/go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.0.1-sdk-v0.44.2.0.20220509210332-5630422e5cdf/baseapp/abci.go:656 +0x17c fp=0xc005775070 sp=0xc0057746b8 pc=0xe3455c
core0         | github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RegisterGRPCServer.func1({0x2070e38, 0xc00be74f30}, {0x1ae56a0, 0xc00bacee48}, 0xc008f5b400, 0xc00b5830e0)
core0         | 	/go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.0.1-sdk-v0.44.2.0.20220509210332-5630422e5cdf/baseapp/grpcserver.go:50 +0x20e fp=0xc0057759a8 sp=0xc005775070 pc=0xe3ba6e
core0         | github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x2070e38, 0xc00be74f30}, {0x1ae56a0, 0xc00bacee48})
core0         | 	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a fp=0xc0057759e8 sp=0xc0057759a8 pc=0xe2e03a
core0         | github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1({0x2070e38, 0xc00be74f30}, {0x1ae56a0, 0xc00bacee48}, 0x18, 0xc008f5b400)
core0         | 	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:33 +0xc8 fp=0xc005775a78 sp=0xc0057759e8 pc=0xe2e1e8
core0         | github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x2070e38, 0xc00be74f30}, {0x1ae56a0, 0xc00bacee48})
core0         | 	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a fp=0xc005775ab8 sp=0xc005775a78 pc=0xe2e03a
core0         | github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x2070e38, 0xc00be74f30}, {0x1ae56a0, 0xc00bacee48}, 0xc002481b01, 0xc002481b58)
core0         | 	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34 +0xbf fp=0xc005775b10 sp=0xc005775ab8 pc=0xe2dedf
core0         | github.com/celestiaorg/celestia-app/x/qgb/types._Query_AttestationRequestByNonce_Handler({0x1b90360, 0xc0001874d0}, {0x2070e38, 0xc00be74f30}, 0xc0083908a0, 0xc00be74f60)
core0         | 	/celestia-app/x/qgb/types/query.pb.go:1556 +0x138 fp=0xc005775b68 sp=0xc005775b10 pc=0x1791e98
core0         | github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RegisterGRPCServer.func2({0x1b90360, 0xc0001874d0}, {0x2070e38, 0xc00be74f30}, 0x1b53d20, 0xc008263a00)
core0         | 	/go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.0.1-sdk-v0.44.2.0.20220509210332-5630422e5cdf/baseapp/grpcserver.go:80 +0xdc fp=0xc005775bc8 sp=0xc005775b68 pc=0xe3b7fc
core0         | google.golang.org/grpc.(*Server).processUnaryRPC(0xc0006ae1c0, {0x208cd10, 0xc000d84900}, 0xc008263a00, 0xc000eac690, 0xc008ce9c38, 0x0)
core0         | 	/go/pkg/mod/google.golang.org/grpc@v1.33.2/server.go:1210 +0xc8f fp=0xc005775e48 sp=0xc005775bc8 pc=0x97e24f
core0         | google.golang.org/grpc.(*Server).handleStream(0xc0006ae1c0, {0x208cd10, 0xc000d84900}, 0xc008263a00, 0x0)
core0         | 	/go/pkg/mod/google.golang.org/grpc@v1.33.2/server.go:1533 +0xa2a fp=0xc005775f68 sp=0xc005775e48 pc=0x981e2a
core0         | google.golang.org/grpc.(*Server).serveStreams.func1.2()
core0         | 	/go/pkg/mod/google.golang.org/grpc@v1.33.2/server.go:871 +0x98 fp=0xc005775fe0 sp=0xc005775f68 pc=0x97bef8
core0         | runtime.goexit()
core0         | 	/usr/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc005775fe8 sp=0xc005775fe0 pc=0x46c901
core0         | created by google.golang.org/grpc.(*Server).serveStreams.func1
core0         | 	/go/pkg/mod/google.golang.org/grpc@v1.33.2/server.go:869 +0x294
core0         | 
core0         | goroutine 1 [chan receive, 32 minutes]:
core0         | github.com/cosmos/cosmos-sdk/server.WaitForQuitSignals()
core0         | 	/go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.0.1-sdk-v0.44.2.0.20220509210332-5630422e5cdf/server/util.go:346 +0x8b
core0         | github.com/cosmos/cosmos-sdk/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x20b0918, 0xc008ce4380}, {0x0, 0x0}, {0x207a140, 0xc008919850}, ...}, ...)
core0         | 	/go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.0.1-sdk-v0.44.2.0.20220509210332-5630422e5cdf/server/start.go:399 +0x126d
core0         | github.com/cosmos/cosmos-sdk/server.StartCmd.func2(0xc008a88000, {0xc000f85860, 0x0, 0x6})
core0         | 	/go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.0.1-sdk-v0.44.2.0.20220509210332-5630422e5cdf/server/start.go:129 +0x193
core0         | github.com/spf13/cobra.(*Command).execute(0xc008a88000, {0xc000f857a0, 0x6, 0x6})
core0         | 	/go/pkg/mod/github.com/spf13/cobra@v1.3.0/command.go:856 +0x60e
core0         | github.com/spf13/cobra.(*Command).ExecuteC(0xc008a22f00)
core0         | 	/go/pkg/mod/github.com/spf13/cobra@v1.3.0/command.go:974 +0x3bc
core0         | github.com/spf13/cobra.(*Command).Execute(...)
core0         | 	/go/pkg/mod/github.com/spf13/cobra@v1.3.0/command.go:902
core0         | github.com/spf13/cobra.(*Command).ExecuteContext(...)
core0         | 	/go/pkg/mod/github.com/spf13/cobra@v1.3.0/command.go:895
core0         | github.com/cosmos/cosmos-sdk/server/cmd.Execute(0xc008a22f00, {0xc008a3c600, 0x13})
core0         | 	/go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.0.1-sdk-v0.44.2.0.20220509210332-5630422e5cdf/server/cmd/execute.go:36 +0x1eb
core0         | main.main()
core0         | 	/celestia-app/cmd/celestia-appd/main.go:34 +0x105
core0         | 
core0         | goroutine 41 [select]:
core0         | github.com/tendermint/tendermint/rpc/jsonrpc/server.(*wsConnection).writeRoutine(0xc000dd4000)
core0         | 	/go/pkg/mod/github.com/celestiaorg/celestia-core@v1.0.1-tm-v0.34.16/rpc/jsonrpc/server/ws_handler.go:417 +0x1d9
core0         | github.com/tendermint/tendermint/rpc/jsonrpc/server.(*wsConnection).OnStart(0xc000dd4000)
core0         | 	/go/pkg/mod/github.com/celestiaorg/celestia-core@v1.0.1-tm-v0.34.16/rpc/jsonrpc/server/ws_handler.go:228 +0x9c
core0         | github.com/tendermint/tendermint/libs/service.(*BaseService).Start(0xc000dd4000)
core0         | 	/go/pkg/mod/github.com/celestiaorg/celestia-core@v1.0.1-tm-v0.34.16/libs/service/service.go:140 +0x243
core0         | github.com/tendermint/tendermint/rpc/jsonrpc/server.(*WebsocketManager).WebsocketHandler(0xc00067a630, {0x204f0a0, 0xc000d74240}, 0xc000dac160)
core0         | 	/go/pkg/mod/github.com/celestiaorg/celestia-core@v1.0.1-tm-v0.34.16/rpc/jsonrpc/server/ws_handler.go:91 +0x2e5
core0         | net/http.HandlerFunc.ServeHTTP(0xc000694380, {0x204f0a0, 0xc000d74240}, 0xc008f008a0)
core0         | 	/usr/local/go/src/net/http/server.go:2047 +0x2f
core0         | net/http.(*ServeMux).ServeHTTP(0x160, {0x204f0a0, 0xc000d74240}, 0xc000da8000)
core0         | 	/usr/local/go/src/net/http/server.go:2425 +0x149
core0         | github.com/tendermint/tendermint/rpc/jsonrpc/server.maxBytesHandler.ServeHTTP({{0x2030aa0, 0xc000101440}, 0x410a07}, {0x204f0a0, 0xc000d74240}, 0xc000da8000)
core0         | 	/go/pkg/mod/github.com/celestiaorg/celestia-core@v1.0.1-tm-v0.34.16/rpc/jsonrpc/server/http_server.go:236 +0x122
core0         | github.com/tendermint/tendermint/rpc/jsonrpc/server.RecoverAndLogHandler.func1({0x204f700, 0xc000d960e0}, 0xc000da8000)
core0         | 	/go/pkg/mod/github.com/celestiaorg/celestia-core@v1.0.1-tm-v0.34.16/rpc/jsonrpc/server/http_server.go:209 +0x383
core0         | net/http.HandlerFunc.ServeHTTP(0x0, {0x204f700, 0xc000d960e0}, 0x46a72e)
core0         | 	/usr/local/go/src/net/http/server.go:2047 +0x2f
core0         | net/http.serverHandler.ServeHTTP({0xc0090782a0}, {0x204f700, 0xc000d960e0}, 0xc000da8000)
core0         | 	/usr/local/go/src/net/http/server.go:2879 +0x43b
core0         | net/http.(*conn).serve(0xc000f7c000, {0x2070e38, 0xc009078180})
core0         | 	/usr/local/go/src/net/http/server.go:1930 +0xb08
core0         | created by net/http.(*Server).Serve
core0         | 	/usr/local/go/src/net/http/server.go:3034 +0x4e8
core0         | 
core0         | goroutine 22 [select, 33 minutes]:
core0         | github.com/desertbit/timer.timerRoutine()
core0         | 	/go/pkg/mod/github.com/desertbit/timer@v0.0.0-20180107155436-c41aec40b27f/timers.go:119 +0xc5
core0         | created by github.com/desertbit/timer.init.0
core0         | 	/go/pkg/mod/github.com/desertbit/timer@v0.0.0-20180107155436-c41aec40b27f/timers.go:15 +0x25
core0         | 
core0         | goroutine 23 [chan receive, 33 minutes]:
core0         | github.com/rjeczalik/notify.(*nonrecursiveTree).dispatch(0xc0010d9620, 0xc00004b000)
core0         | 	/go/pkg/mod/github.com/rjeczalik/notify@v0.9.1/tree_nonrecursive.go:36 +0x49
core0         | created by github.com/rjeczalik/notify.newNonrecursiveTree
core0         | 	/go/pkg/mod/github.com/rjeczalik/notify@v0.9.1/tree_nonrecursive.go:29 +0x16f
core0         | 
core0         | goroutine 24 [chan receive, 33 minutes]:
core0         | github.com/rjeczalik/notify.(*nonrecursiveTree).internal(0xc0010d9620, 0xc0010d95c0)
core0         | 	/go/pkg/mod/github.com/rjeczalik/notify@v0.9.1/tree_nonrecursive.go:81 +0x5a
core0         | created by github.com/rjeczalik/notify.newNonrecursiveTree
core0         | 	/go/pkg/mod/github.com/rjeczalik/notify@v0.9.1/tree_nonrecursive.go:30 +0x1c8
core0         | 
core0         | goroutine 53 [chan receive]:
core0         | github.com/rcrowley/go-metrics.(*meterArbiter).tick(0x2f8cb60)
core0         | 	/go/pkg/mod/github.com/rcrowley/go-metrics@v0.0.0-20200313005456-10cdbea86bc0/meter.go:239 +0x2a
core0         | created by github.com/rcrowley/go-metrics.NewMeter
core0         | 	/go/pkg/mod/github.com/rcrowley/go-metrics@v0.0.0-20200313005456-10cdbea86bc0/meter.go:46 +0xd8
core0         | 
core0         | goroutine 25 [select, 7 minutes]:
core0         | github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc008a1fc00)
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_compaction.go:91 +0x15e
core0         | created by github.com/syndtr/goleveldb/leveldb.openDB
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:148 +0x4ef
core0         | 
core0         | goroutine 55 [select]:
core0         | github.com/syndtr/goleveldb/leveldb.(*session).refLoop(0xc00014cf00)
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/session_util.go:189 +0x5bb
core0         | created by github.com/syndtr/goleveldb/leveldb.newSession
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/session.go:93 +0x2de
core0         | 
core0         | goroutine 26 [select]:
core0         | github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc008a1fc00)
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_state.go:101 +0xae
core0         | created by github.com/syndtr/goleveldb/leveldb.openDB
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:149 +0x531
core0         | 
core0         | goroutine 27 [select, 7 minutes]:
core0         | github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc008a1fc00)
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_compaction.go:836 +0x685
core0         | created by github.com/syndtr/goleveldb/leveldb.openDB
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:155 +0x5a7
core0         | 
core0         | goroutine 28 [select, 7 minutes]:
core0         | github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc008a1fc00)
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_compaction.go:773 +0x119
core0         | created by github.com/syndtr/goleveldb/leveldb.openDB
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:156 +0x5e9
core0         | 
core0         | goroutine 29 [select, 3 minutes]:
core0         | github.com/syndtr/goleveldb/leveldb.(*session).refLoop(0xc00014c3c0)
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/session_util.go:189 +0x5bb
core0         | created by github.com/syndtr/goleveldb/leveldb.newSession
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/session.go:93 +0x2de
core0         | 
core0         | goroutine 30 [select, 32 minutes]:
core0         | github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc008a1e000)
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_compaction.go:91 +0x15e
core0         | created by github.com/syndtr/goleveldb/leveldb.openDB
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:148 +0x4ef
core0         | 
core0         | goroutine 31 [select]:
core0         | github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc008a1e000)
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_state.go:101 +0xae
core0         | created by github.com/syndtr/goleveldb/leveldb.openDB
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:149 +0x531
core0         | 
core0         | goroutine 32 [select, 32 minutes]:
core0         | github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc008a1e000)
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_compaction.go:836 +0x685
core0         | created by github.com/syndtr/goleveldb/leveldb.openDB
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:155 +0x5a7
core0         | 
core0         | goroutine 33 [select, 32 minutes]:
core0         | github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc008a1e000)
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_compaction.go:773 +0x119
core0         | created by github.com/syndtr/goleveldb/leveldb.openDB
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:156 +0x5e9
core0         | 
core0         | goroutine 82 [select, 3 minutes]:
core0         | github.com/syndtr/goleveldb/leveldb.(*session).refLoop(0xc00014d0e0)
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/session_util.go:189 +0x5bb
core0         | created by github.com/syndtr/goleveldb/leveldb.newSession
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/session.go:93 +0x2de
core0         | 
core0         | goroutine 83 [select, 32 minutes]:
core0         | github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc008a1e1c0)
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_compaction.go:91 +0x15e
core0         | created by github.com/syndtr/goleveldb/leveldb.openDB
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:148 +0x4ef
core0         | 
core0         | goroutine 84 [select]:
core0         | github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc008a1e1c0)
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_state.go:101 +0xae
core0         | created by github.com/syndtr/goleveldb/leveldb.openDB
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:149 +0x531
core0         | 
core0         | goroutine 85 [select, 32 minutes]:
core0         | github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc008a1e1c0)
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_compaction.go:836 +0x685
core0         | created by github.com/syndtr/goleveldb/leveldb.openDB
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:155 +0x5a7
core0         | 
core0         | goroutine 86 [select, 32 minutes]:
core0         | github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc008a1e1c0)
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_compaction.go:773 +0x119
core0         | created by github.com/syndtr/goleveldb/leveldb.openDB
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:156 +0x5e9
core0         | 
core0         | goroutine 87 [select, 3 minutes]:
core0         | github.com/syndtr/goleveldb/leveldb.(*session).refLoop(0xc00014d1d0)
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/session_util.go:189 +0x5bb
core0         | created by github.com/syndtr/goleveldb/leveldb.newSession
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/session.go:93 +0x2de
core0         | 
core0         | goroutine 88 [select, 32 minutes]:
core0         | github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc008a1e380)
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_compaction.go:91 +0x15e
core0         | created by github.com/syndtr/goleveldb/leveldb.openDB
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:148 +0x4ef
core0         | 
core0         | goroutine 89 [select]:
core0         | github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc008a1e380)
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_state.go:101 +0xae
core0         | created by github.com/syndtr/goleveldb/leveldb.openDB
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:149 +0x531
core0         | 
core0         | goroutine 90 [select, 32 minutes]:
core0         | github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc008a1e380)
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_compaction.go:836 +0x685
core0         | created by github.com/syndtr/goleveldb/leveldb.openDB
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:155 +0x5a7
core0         | 
core0         | goroutine 91 [select, 32 minutes]:
core0         | github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc008a1e380)
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_compaction.go:773 +0x119
core0         | created by github.com/syndtr/goleveldb/leveldb.openDB
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:156 +0x5e9
core0         | 
core0         | goroutine 92 [select, 32 minutes]:
core0         | github.com/tendermint/tendermint/proxy.(*multiAppConn).killTMOnClientError(0xc008f9ac30)
core0         | 	/go/pkg/mod/github.com/celestiaorg/celestia-core@v1.0.1-tm-v0.34.16/proxy/multi_app_conn.go:138 +0x128
core0         | created by github.com/tendermint/tendermint/proxy.(*multiAppConn).OnStart
core0         | 	/go/pkg/mod/github.com/celestiaorg/celestia-core@v1.0.1-tm-v0.34.16/proxy/multi_app_conn.go:118 +0x32f
core0         | 
core0         | goroutine 93 [chan receive]:
core0         | github.com/tendermint/tendermint/libs/pubsub.(*Server).loop(0xc008a3f810, {0xc000159200, 0xc000159290})
core0         | 	/go/pkg/mod/github.com/celestiaorg/celestia-core@v1.0.1-tm-v0.34.16/libs/pubsub/pubsub.go:324 +0x9a
core0         | created by github.com/tendermint/tendermint/libs/pubsub.(*Server).OnStart
core0         | 	/go/pkg/mod/github.com/celestiaorg/celestia-core@v1.0.1-tm-v0.34.16/libs/pubsub/pubsub.go:310 +0xaf
core0         | 
core0         | goroutine 94 [select, 3 minutes]:
core0         | github.com/syndtr/goleveldb/leveldb.(*session).refLoop(0xc00014d2c0)
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/session_util.go:189 +0x5bb
core0         | created by github.com/syndtr/goleveldb/leveldb.newSession
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/session.go:93 +0x2de
core0         | 
core0         | goroutine 95 [select, 32 minutes]:
core0         | github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc008a1e540)
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_compaction.go:91 +0x15e
core0         | created by github.com/syndtr/goleveldb/leveldb.openDB
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:148 +0x4ef
core0         | 
core0         | goroutine 96 [select]:
core0         | github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc008a1e540)
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_state.go:101 +0xae
core0         | created by github.com/syndtr/goleveldb/leveldb.openDB
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:149 +0x531
core0         | 
core0         | goroutine 97 [select, 32 minutes]:
core0         | github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc008a1e540)
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_compaction.go:836 +0x685
core0         | created by github.com/syndtr/goleveldb/leveldb.openDB
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:155 +0x5a7
core0         | 
core0         | goroutine 98 [select, 32 minutes]:
core0         | github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc008a1e540)
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_compaction.go:773 +0x119
core0         | created by github.com/syndtr/goleveldb/leveldb.openDB
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:156 +0x5e9
core0         | 
core0         | goroutine 99 [chan receive]:
core0         | github.com/tendermint/tendermint/state/txindex.(*IndexerService).OnStart.func1()
core0         | 	/go/pkg/mod/github.com/celestiaorg/celestia-core@v1.0.1-tm-v0.34.16/state/txindex/indexer_service.go:60 +0xab
core0         | created by github.com/tendermint/tendermint/state/txindex.(*IndexerService).OnStart
core0         | 	/go/pkg/mod/github.com/celestiaorg/celestia-core@v1.0.1-tm-v0.34.16/state/txindex/indexer_service.go:58 +0x1d7
core0         | 
core0         | goroutine 196 [select, 3 minutes]:
core0         | github.com/syndtr/goleveldb/leveldb.(*session).refLoop(0xc00063de00)
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/session_util.go:189 +0x5bb
core0         | created by github.com/syndtr/goleveldb/leveldb.newSession
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/session.go:93 +0x2de
core0         | 
core0         | goroutine 197 [select, 32 minutes]:
core0         | github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc000647dc0)
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_compaction.go:91 +0x15e
core0         | created by github.com/syndtr/goleveldb/leveldb.openDB
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:148 +0x4ef
core0         | 
core0         | goroutine 175 [select]:
core0         | google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc008a674a0, 0x1)
core0         | 	/go/pkg/mod/google.golang.org/grpc@v1.33.2/internal/transport/controlbuf.go:395 +0x11b
core0         | google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000521f80)
core0         | 	/go/pkg/mod/google.golang.org/grpc@v1.33.2/internal/transport/controlbuf.go:515 +0x85
core0         | google.golang.org/grpc/internal/transport.newHTTP2Server.func2()
core0         | 	/go/pkg/mod/google.golang.org/grpc@v1.33.2/internal/transport/http2_server.go:291 +0xc6
core0         | created by google.golang.org/grpc/internal/transport.newHTTP2Server
core0         | 	/go/pkg/mod/google.golang.org/grpc@v1.33.2/internal/transport/http2_server.go:288 +0x148f
core0         | 
core0         | goroutine 176 [select, 32 minutes]:
core0         | google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc000d84900)
core0         | 	/go/pkg/mod/google.golang.org/grpc@v1.33.2/internal/transport/http2_server.go:979 +0x245
core0         | created by google.golang.org/grpc/internal/transport.newHTTP2Server
core0         | 	/go/pkg/mod/google.golang.org/grpc@v1.33.2/internal/transport/http2_server.go:299 +0x14d7
core0         | 
core0         | goroutine 198 [select]:
core0         | github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc000647dc0)
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_state.go:101 +0xae
core0         | created by github.com/syndtr/goleveldb/leveldb.openDB
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:149 +0x531
core0         | 
core0         | goroutine 199 [select, 32 minutes]:
core0         | github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc000647dc0)
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_compaction.go:836 +0x685
core0         | created by github.com/syndtr/goleveldb/leveldb.openDB
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:155 +0x5a7
core0         | 
core0         | goroutine 200 [select, 32 minutes]:
core0         | github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc000647dc0)
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_compaction.go:773 +0x119
core0         | created by github.com/syndtr/goleveldb/leveldb.openDB
core0         | 	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:156 +0x5e9
core0         | 
core0         | goroutine 201 [IO wait, 32 minutes]:
core0         | internal/poll.runtime_pollWait(0x7f43c89dc178, 0x72)
core0         | 	/usr/local/go/src/runtime/netpoll.go:303 +0x85
core0         | internal/poll.(*pollDesc).wait(0xc008b42400, 0x4, 0x0)
core0         | 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
core0         | internal/poll.(*pollDesc).waitRead(...)
core0         | 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
core0         | internal/poll.(*FD).Accept(0xc008b42400)
core0         | 	/usr/local/go/src/internal/poll/fd_unix.go:402 +0x22c
core0         | net.(*netFD).accept(0xc008b42400)
core0         | 	/usr/local/go/src/net/fd_unix.go:173 +0x35
core0         | net.(*TCPListener).accept(0xc00000fbf0)
core0         | 	/usr/local/go/src/net/tcpsock_posix.go:140 +0x28
core0         | net.(*TCPListener).Accept(0xc00000fbf0)
core0         | 	/usr/local/go/src/net/tcpsock.go:262 +0x3d
core0         | net/http.(*Server).Serve(0xc00057e000, {0x204f4f0, 0xc00000fbf0})
core0         | 	/usr/local/go/src/net/http/server.go:3002 +0x394
core0         | net/http.(*Server).ListenAndServe(0xc00057e000)
core0         | 	/usr/local/go/src/net/http/server.go:2931 +0x7d
core0         | net/http.ListenAndServe(...)
core0         | 	/usr/local/go/src/net/http/server.go:3185
core0         | github.com/tendermint/tendermint/node.NewNode.func1()
core0         | 	/go/pkg/mod/github.com/celestiaorg/celestia-core@v1.0.1-tm-v0.34.16/node/node.go:853 +0x139
core0         | created by github.com/tendermint/tendermint/node.NewNode
core0         | 	/go/pkg/mod/github.com/celestiaorg/celestia-core@v1.0.1-tm-v0.34.16/node/node.go:851 +0x1545
core0         | 
core0         | goroutine 202 [IO wait]:
core0         | internal/poll.runtime_pollWait(0x7f43c89dc358, 0x72)
core0         | 	/usr/local/go/src/runtime/netpoll.go:303 +0x85
core0         | internal/poll.(*pollDesc).wait(0xc0022bae80, 0x2, 0x0)
core0         | 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
core0         | internal/poll.(*pollDesc).waitRead(...)
core0         | 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
core0         | internal/poll.(*FD).Accept(0xc0022bae80)
core0         | 	/usr/local/go/src/internal/poll/fd_unix.go:402 +0x22c
core0         | net.(*netFD).accept(0xc0022bae80)
core0         | 	/usr/local/go/src/net/fd_unix.go:173 +0x35
core0         | net.(*TCPListener).accept(0xc0006ec060)
core0         | 	/usr/local/go/src/net/tcpsock_posix.go:140 +0x28
core0         | net.(*TCPListener).Accept(0xc0006ec060)
core0         | 	/usr/local/go/src/net/tcpsock.go:262 +0x3d
core0         | golang.org/x/net/netutil.(*limitListener).Accept(0xc0006dcf90)
core0         | 	/go/pkg/mod/golang.org/x/net@v0.0.0-20211108170745-6635138e15ea/netutil/listen.go:48 +0x3e
core0         | net/http.(*Server).Serve(0xc000d96000, {0x204f190, 0xc0006dcf90})
core0         | 	/usr/local/go/src/net/http/server.go:3002 +0x394
core0         | github.com/tendermint/tendermint/rpc/jsonrpc/server.Serve({0x204f190, 0xc0006dcf90}, {0x2030aa0, 0xc000101440}, {0x2071b58, 0xc0006a86c0}, 0xc008ee8ed0)
core0         | 	/go/pkg/mod/github.com/celestiaorg/celestia-core@v1.0.1-tm-v0.34.16/rpc/jsonrpc/server/http_server.go:61 +0x190
core0         | github.com/tendermint/tendermint/node.(*Node).startRPC.func3()
core0         | 	/go/pkg/mod/github.com/celestiaorg/celestia-core@v1.0.1-tm-v0.34.16/node/node.go:1129 +0x45
core0         | created by github.com/tendermint/tendermint/node.(*Node).startRPC
core0         | 	/go/pkg/mod/github.com/celestiaorg/celestia-core@v1.0.1-tm-v0.34.16/node/node.go:1128 +0xbd1
core0         | 
core0         | goroutine 203 [IO wait, 32 minutes]:
core0         | internal/poll.runtime_pollWait(0x7f43c89dc268, 0x72)
core0         | 	/usr/local/go/src/runtime/netpoll.go:303 +0x85
core0         | internal/poll.(*pollDesc).wait(0xc0022baf00, 0x2, 0x0)
core0         | 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
core0         | internal/poll.(*pollDesc).waitRead(...)
core0         | 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
core0         | internal/poll.(*FD).Accept(0xc0022baf00)
@rach-id rach-id added the C: QGB label Jul 4, 2022
@rach-id rach-id moved this to TODO in Celestia Node Jul 4, 2022
@rach-id rach-id mentioned this issue Jul 4, 2022
Closed
90 tasks
@evan-forbes
Copy link
Member

per sync discussion, this hasn't occurred since, so we're optimistically closing

Repository owner moved this from TODO to Done in Celestia Node Nov 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
No open projects
Celestia Node
Archived in project
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rach-id @evan-forbes