Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix a race in tipset cache usage #4282

Merged
merged 1 commit into from
Oct 10, 2020
Merged

fix a race in tipset cache usage #4282

merged 1 commit into from
Oct 10, 2020

Conversation

Stebalien
Copy link
Member

This tipset cache is shared between multiple services and is called from multiple places.

@Stebalien
Copy link
Member Author

logs 
==================

WARNING: DATA RACE

Write at 0x00c001963458 by goroutine 877:

github.com/filecoin-project/lotus/chain/events.(*tipSetCache).add()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/tscache.go:59 +0x359

github.com/filecoin-project/lotus/chain/events.(*heightEvents).headChangeAt()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events_height.go:84 +0x724

github.com/filecoin-project/lotus/chain/events.(*Events).headChange()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events.go:175 +0x20a

github.com/filecoin-project/lotus/chain/events.(*Events).listenHeadChangesOnce()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events.go:154 +0x7a4

github.com/filecoin-project/lotus/chain/events.(*Events).listenHeadChanges()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events.go:94 +0x11e


Previous read at 0x00c001963458 by goroutine 1042:

github.com/filecoin-project/lotus/chain/events.(*tipSetCache).best()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/tscache.go:127 +0x64

github.com/filecoin-project/lotus/chain/events.(*hcEvents).onHeadChanged()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events_called.go:310 +0xe4

github.com/filecoin-project/lotus/chain/events.(*watcherEvents).StateChanged()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events_called.go:443 +0x137

github.com/filecoin-project/lotus/markets/storageadapter.(*ProviderNodeAdapter).OnDealExpiredOrSlashed()

/home/steb/src/github.com/filecoin-project/lotus/markets/storageadapter/provider.go:475 +0x9fb

github.com/filecoin-project/go-fil-markets/storagemarket/impl/providerstates.WaitForDealCompletion()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/go-fil-markets@v0.7.1/storagemarket/impl/providerstates/provider_states.go:454 +0x201

runtime.call512()

/usr/lib/go/src/runtime/asm_amd64.s:544 +0x5b

reflect.Value.Call()

/usr/lib/go/src/reflect/value.go:336 +0xd8

github.com/filecoin-project/go-statemachine/fsm.fsmHandler.handler.func1()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/go-statemachine@v0.0.0-20200925024713-05bd7c71fbfe/fsm/fsm.go:170 +0x484

reflect.callReflect()

/usr/lib/go/src/reflect/value.go:564 +0x5c5

reflect.makeFuncStub()

/usr/lib/go/src/reflect/asm_amd64.s:20 +0x41

reflect.Value.Call()

/usr/lib/go/src/reflect/value.go:336 +0xd8

github.com/filecoin-project/go-statemachine.(*StateMachine).run.func3()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/go-statemachine@v0.0.0-20200925024713-05bd7c71fbfe/machine.go:102 +0x27c


Goroutine 877 (running) created at:

github.com/filecoin-project/lotus/chain/events.NewEvents()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events.go:83 +0x7c4

github.com/filecoin-project/lotus/markets/storageadapter.NewProviderNodeAdapter.func1()

/home/steb/src/github.com/filecoin-project/lotus/markets/storageadapter/provider.go:64 +0xc8

runtime.call64()

/usr/lib/go/src/runtime/asm_amd64.s:541 +0x3d

reflect.Value.Call()

/usr/lib/go/src/reflect/value.go:336 +0xd8

github.com/filecoin-project/lotus/node.as.func2()

/home/steb/src/github.com/filecoin-project/lotus/node/options.go:140 +0xb5

reflect.callReflect()

/usr/lib/go/src/reflect/value.go:564 +0x5c5

reflect.makeFuncStub()

/usr/lib/go/src/reflect/asm_amd64.s:20 +0x41

reflect.Value.Call()

/usr/lib/go/src/reflect/value.go:336 +0xd8

go.uber.org/dig.defaultInvoker()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:284 +0x7a

go.uber.org/dig.(*node).Call()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:710 +0x2d0

go.uber.org/dig.paramSingle.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x39a

go.uber.org/dig.(*paramSingle).Build()

:1 +0xe7

go.uber.org/dig.paramList.BuildList()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0xf7

go.uber.org/dig.(*node).Call()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0x155

go.uber.org/dig.paramSingle.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x39a

go.uber.org/dig.(*paramSingle).Build()

:1 +0xe7

go.uber.org/dig.paramList.BuildList()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0xf7

go.uber.org/dig.(*node).Call()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0x155

go.uber.org/dig.paramSingle.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x39a

go.uber.org/dig.(*paramSingle).Build()

:1 +0xe7

go.uber.org/dig.paramList.BuildList()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0xf7

go.uber.org/dig.(*Container).Invoke()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:432 +0x372

go.uber.org/fx.(*App).executeInvokes()

/home/steb/.local/share/go/pkg/mod/go.uber.org/fx@v1.9.0/app.go:547 +0x493

go.uber.org/fx.New()

/home/steb/.local/share/go/pkg/mod/go.uber.org/fx@v1.9.0/app.go:345 +0x83b

github.com/filecoin-project/lotus/node.New()

/home/steb/src/github.com/filecoin-project/lotus/node/builder.go:570 +0x64e

github.com/filecoin-project/lotus/node/test.CreateTestStorageNode()

/home/steb/src/github.com/filecoin-project/lotus/node/test/builder.go:107 +0x11b3

github.com/filecoin-project/lotus/node/test.mockSbBuilderOpts()

/home/steb/src/github.com/filecoin-project/lotus/node/test/builder.go:445 +0x24ad

github.com/filecoin-project/lotus/node/test.MockSbBuilder()

/home/steb/src/github.com/filecoin-project/lotus/node/test/builder.go:149 +0xa8

github.com/filecoin-project/lotus/api/test.TestDealFlow()

/home/steb/src/github.com/filecoin-project/lotus/api/test/deals.go:51 +0x148

github.com/filecoin-project/lotus/node_test.TestAPIDealFlow.func1()

/home/steb/src/github.com/filecoin-project/lotus/node/node_test.go:42 +0x54

testing.tRunner()

/usr/lib/go/src/testing/testing.go:1127 +0x202


Goroutine 1042 (finished) created at:

github.com/filecoin-project/go-statemachine.(*StateMachine).run()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/go-statemachine@v0.0.0-20200925024713-05bd7c71fbfe/machine.go:100 +0x7ab


==================

WARNING: DATA RACE

Write at 0x00c00055ff58 by goroutine 397:

github.com/filecoin-project/lotus/chain/events.(*tipSetCache).add()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/tscache.go:59 +0x359

github.com/filecoin-project/lotus/chain/events.(*heightEvents).headChangeAt()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events_height.go:84 +0x724

github.com/filecoin-project/lotus/chain/events.(*Events).headChange()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events.go:175 +0x20a

github.com/filecoin-project/lotus/chain/events.(*Events).listenHeadChangesOnce()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events.go:154 +0x7a4

github.com/filecoin-project/lotus/chain/events.(*Events).listenHeadChanges()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events.go:94 +0x11e


Previous read at 0x00c00055ff58 by goroutine 250:

github.com/filecoin-project/lotus/chain/events.(*tipSetCache).best()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/tscache.go:127 +0x64

github.com/filecoin-project/lotus/chain/events.(*hcEvents).onHeadChanged()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events_called.go:310 +0xe4

github.com/filecoin-project/lotus/chain/events.(*watcherEvents).StateChanged()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events_called.go:443 +0x137

github.com/filecoin-project/lotus/markets/storageadapter.(*ClientNodeAdapter).OnDealExpiredOrSlashed()

/home/steb/src/github.com/filecoin-project/lotus/markets/storageadapter/client.go:406 +0x9e4

github.com/filecoin-project/go-fil-markets/storagemarket/impl/clientstates.WaitForDealCompletion()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/go-fil-markets@v0.7.1/storagemarket/impl/clientstates/client_states.go:270 +0x193

runtime.call512()

/usr/lib/go/src/runtime/asm_amd64.s:544 +0x5b

reflect.Value.Call()

/usr/lib/go/src/reflect/value.go:336 +0xd8

github.com/filecoin-project/go-statemachine/fsm.fsmHandler.handler.func1()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/go-statemachine@v0.0.0-20200925024713-05bd7c71fbfe/fsm/fsm.go:170 +0x484

reflect.callReflect()

/usr/lib/go/src/reflect/value.go:564 +0x5c5

reflect.makeFuncStub()

/usr/lib/go/src/reflect/asm_amd64.s:20 +0x41

reflect.Value.Call()

/usr/lib/go/src/reflect/value.go:336 +0xd8

github.com/filecoin-project/go-statemachine.(*StateMachine).run.func3()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/go-statemachine@v0.0.0-20200925024713-05bd7c71fbfe/machine.go:102 +0x27c


Goroutine 397 (running) created at:

github.com/filecoin-project/lotus/chain/events.NewEvents()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events.go:83 +0x7c4

github.com/filecoin-project/lotus/markets/storageadapter.NewClientNodeAdapter()

/home/steb/src/github.com/filecoin-project/lotus/markets/storageadapter/client.go:61 +0x144

runtime.call512()

/usr/lib/go/src/runtime/asm_amd64.s:544 +0x5b

reflect.Value.Call()

/usr/lib/go/src/reflect/value.go:336 +0xd8

github.com/filecoin-project/lotus/node.as.func2()

/home/steb/src/github.com/filecoin-project/lotus/node/options.go:140 +0xb5

reflect.callReflect()

/usr/lib/go/src/reflect/value.go:564 +0x5c5

reflect.makeFuncStub()

/usr/lib/go/src/reflect/asm_amd64.s:20 +0x41

reflect.Value.Call()

/usr/lib/go/src/reflect/value.go:336 +0xd8

go.uber.org/dig.defaultInvoker()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:284 +0x7a

go.uber.org/dig.(*node).Call()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:710 +0x2d0

go.uber.org/dig.paramSingle.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x39a

go.uber.org/dig.(*paramSingle).Build()

:1 +0xe7

go.uber.org/dig.paramList.BuildList()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0xf7

go.uber.org/dig.(*node).Call()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0x155

go.uber.org/dig.paramSingle.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x39a

go.uber.org/dig.(*paramSingle).Build()

:1 +0xe7

go.uber.org/dig.paramObjectField.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:384 +0x1c3

go.uber.org/dig.paramObject.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:311 +0x160

go.uber.org/dig.(*paramObject).Build()

:1 +0xe7

go.uber.org/dig.paramObjectField.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:384 +0x1c3

go.uber.org/dig.paramObject.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:311 +0x160

go.uber.org/dig.(*paramObject).Build()

:1 +0xe7

go.uber.org/dig.paramList.BuildList()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0xf7

go.uber.org/dig.(*Container).Invoke()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:432 +0x372

go.uber.org/fx.(*App).executeInvokes()

/home/steb/.local/share/go/pkg/mod/go.uber.org/fx@v1.9.0/app.go:547 +0x493

go.uber.org/fx.New()

/home/steb/.local/share/go/pkg/mod/go.uber.org/fx@v1.9.0/app.go:345 +0x83b

github.com/filecoin-project/lotus/node.New()

/home/steb/src/github.com/filecoin-project/lotus/node/builder.go:570 +0x64e

github.com/filecoin-project/lotus/node/test.mockSbBuilderOpts()

/home/steb/src/github.com/filecoin-project/lotus/node/test/builder.go:396 +0x184e

github.com/filecoin-project/lotus/node/test.MockSbBuilder()

/home/steb/src/github.com/filecoin-project/lotus/node/test/builder.go:149 +0xa8

github.com/filecoin-project/lotus/api/test.TestDealFlow()

/home/steb/src/github.com/filecoin-project/lotus/api/test/deals.go:51 +0x148

github.com/filecoin-project/lotus/node_test.TestAPIDealFlow.func1()

/home/steb/src/github.com/filecoin-project/lotus/node/node_test.go:42 +0x54

testing.tRunner()

/usr/lib/go/src/testing/testing.go:1127 +0x202


Goroutine 250 (finished) created at:

github.com/filecoin-project/go-statemachine.(*StateMachine).run()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/go-statemachine@v0.0.0-20200925024713-05bd7c71fbfe/machine.go:100 +0x7ab


==================

WARNING: DATA RACE

Write at 0x00c001f83118 by goroutine 898:

github.com/filecoin-project/lotus/chain/events.(*tipSetCache).add()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/tscache.go:59 +0x359

github.com/filecoin-project/lotus/chain/events.(*heightEvents).headChangeAt()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events_height.go:84 +0x724

github.com/filecoin-project/lotus/chain/events.(*Events).headChange()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events.go:175 +0x20a

github.com/filecoin-project/lotus/chain/events.(*Events).listenHeadChangesOnce()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events.go:154 +0x7a4

github.com/filecoin-project/lotus/chain/events.(*Events).listenHeadChanges()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events.go:94 +0x11e


Previous read at 0x00c001f83118 by goroutine 706:

??()

-:0 +0xffffffffffffffff

github.com/ipfs/go-ipfs-ds-help.NewKeyFromBinary()

/home/steb/.local/share/go/pkg/mod/github.com/ipfs/go-ipfs-ds-help@v1.0.0/key.go:16 +0x16c

github.com/ipfs/go-ipfs-ds-help.MultihashToDsKey()

/home/steb/.local/share/go/pkg/mod/github.com/ipfs/go-ipfs-ds-help@v1.0.0/key.go:30 +0x8d

github.com/ipfs/go-ipfs-blockstore.(*blockstore).Get()

/home/steb/.local/share/go/pkg/mod/github.com/ipfs/go-ipfs-blockstore@v1.0.1/blockstore.go:124 +0x56

github.com/ipfs/go-ipfs-blockstore.(*idstore).Get()

/home/steb/.local/share/go/pkg/mod/github.com/ipfs/go-ipfs-blockstore@v1.0.1/idstore.go:62 +0x136

github.com/ipfs/go-ipfs-blockstore.(*arccache).Get()

/home/steb/.local/share/go/pkg/mod/github.com/ipfs/go-ipfs-blockstore@v1.0.1/arc_cache.go:118 +0xd8

github.com/ipfs/go-ipfs-blockstore.(*bloomcache).Get()

/home/steb/.local/share/go/pkg/mod/github.com/ipfs/go-ipfs-blockstore@v1.0.1/bloom_cache.go:159 +0xa6

github.com/ipfs/go-ipfs-blockstore.(*idstore).Get()

/home/steb/.local/share/go/pkg/mod/github.com/ipfs/go-ipfs-blockstore@v1.0.1/idstore.go:62 +0x136

github.com/ipfs/go-ipld-cbor.(*BasicIpldStore).Get()

/home/steb/.local/share/go/pkg/mod/github.com/ipfs/go-ipld-cbor@v0.0.5-0.20200428170625-a0bd04d3cbdf/store.go:44 +0x8a

github.com/filecoin-project/specs-actors/actors/util/adt.(*wstore).Get()

:1 +0xb6

github.com/filecoin-project/go-amt-ipld/v2.LoadAMT()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/go-amt-ipld/v2@v2.1.1-0.20201006184820-924ee87a1349/amt.go:56 +0x101

github.com/filecoin-project/specs-actors/actors/util/adt.AsArray()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/specs-actors@v0.9.12/actors/util/adt/array.go:22 +0xe4

github.com/filecoin-project/lotus/chain/actors/builtin/market.(*state0).States()

/home/steb/src/github.com/filecoin-project/lotus/chain/actors/builtin/market/v0.go:61 +0xc7

github.com/filecoin-project/lotus/chain/stmgr.GetStorageDeal()

/home/steb/src/github.com/filecoin-project/lotus/chain/stmgr/utils.go:325 +0x566

github.com/filecoin-project/lotus/node/impl/full.(*StateAPI).StateMarketStorageDeal()

/home/steb/src/github.com/filecoin-project/lotus/node/impl/full/state.go:658 +0x237

github.com/filecoin-project/lotus/node/impl.(*FullNodeAPI).StateMarketStorageDeal()

:1 +0x8f

github.com/filecoin-project/lotus/api/test.(*TestNode).StateMarketStorageDeal()

:1 +0xaa

github.com/filecoin-project/lotus/markets/storageadapter.(*ProviderNodeAdapter).OnDealExpiredOrSlashed()

/home/steb/src/github.com/filecoin-project/lotus/markets/storageadapter/provider.go:393 +0x28d

github.com/filecoin-project/go-fil-markets/storagemarket/impl/providerstates.WaitForDealCompletion()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/go-fil-markets@v0.7.1/storagemarket/impl/providerstates/provider_states.go:454 +0x201

runtime.call512()

/usr/lib/go/src/runtime/asm_amd64.s:544 +0x5b

reflect.Value.Call()

/usr/lib/go/src/reflect/value.go:336 +0xd8

github.com/filecoin-project/go-statemachine/fsm.fsmHandler.handler.func1()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/go-statemachine@v0.0.0-20200925024713-05bd7c71fbfe/fsm/fsm.go:170 +0x484

reflect.callReflect()

/usr/lib/go/src/reflect/value.go:564 +0x5c5

reflect.makeFuncStub()

/usr/lib/go/src/reflect/asm_amd64.s:20 +0x41

reflect.Value.Call()

/usr/lib/go/src/reflect/value.go:336 +0xd8

github.com/filecoin-project/go-statemachine.(*StateMachine).run.func3()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/go-statemachine@v0.0.0-20200925024713-05bd7c71fbfe/machine.go:102 +0x27c


Goroutine 898 (running) created at:

github.com/filecoin-project/lotus/chain/events.NewEvents()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events.go:83 +0x7c4

github.com/filecoin-project/lotus/markets/storageadapter.NewProviderNodeAdapter.func1()

/home/steb/src/github.com/filecoin-project/lotus/markets/storageadapter/provider.go:64 +0xc8

runtime.call64()

/usr/lib/go/src/runtime/asm_amd64.s:541 +0x3d

reflect.Value.Call()

/usr/lib/go/src/reflect/value.go:336 +0xd8

github.com/filecoin-project/lotus/node.as.func2()

/home/steb/src/github.com/filecoin-project/lotus/node/options.go:140 +0xb5

reflect.callReflect()

/usr/lib/go/src/reflect/value.go:564 +0x5c5

reflect.makeFuncStub()

/usr/lib/go/src/reflect/asm_amd64.s:20 +0x41

reflect.Value.Call()

/usr/lib/go/src/reflect/value.go:336 +0xd8

go.uber.org/dig.defaultInvoker()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:284 +0x7a

go.uber.org/dig.(*node).Call()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:710 +0x2d0

go.uber.org/dig.paramSingle.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x39a

go.uber.org/dig.(*paramSingle).Build()

:1 +0xe7

go.uber.org/dig.paramList.BuildList()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0xf7

go.uber.org/dig.(*node).Call()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0x155

go.uber.org/dig.paramSingle.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x39a

go.uber.org/dig.(*paramSingle).Build()

:1 +0xe7

go.uber.org/dig.paramList.BuildList()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0xf7

go.uber.org/dig.(*node).Call()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0x155

go.uber.org/dig.paramSingle.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x39a

go.uber.org/dig.(*paramSingle).Build()

:1 +0xe7

go.uber.org/dig.paramList.BuildList()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0xf7

go.uber.org/dig.(*Container).Invoke()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:432 +0x372

go.uber.org/fx.(*App).executeInvokes()

/home/steb/.local/share/go/pkg/mod/go.uber.org/fx@v1.9.0/app.go:547 +0x493

go.uber.org/fx.New()

/home/steb/.local/share/go/pkg/mod/go.uber.org/fx@v1.9.0/app.go:345 +0x83b

github.com/filecoin-project/lotus/node.New()

/home/steb/src/github.com/filecoin-project/lotus/node/builder.go:570 +0x64e

github.com/filecoin-project/lotus/node/test.CreateTestStorageNode()

/home/steb/src/github.com/filecoin-project/lotus/node/test/builder.go:107 +0x11b3

github.com/filecoin-project/lotus/node/test.mockBuilderOpts()

/home/steb/src/github.com/filecoin-project/lotus/node/test/builder.go:281 +0x2504

github.com/filecoin-project/lotus/node/test.Builder()

/home/steb/src/github.com/filecoin-project/lotus/node/test/builder.go:145 +0xa8

github.com/filecoin-project/lotus/api/test.TestDealFlow()

/home/steb/src/github.com/filecoin-project/lotus/api/test/deals.go:51 +0x148

github.com/filecoin-project/lotus/node_test.TestAPIDealFlowReal.func3()

/home/steb/src/github.com/filecoin-project/lotus/node/node_test.go:78 +0x54

testing.tRunner()

/usr/lib/go/src/testing/testing.go:1127 +0x202


Goroutine 706 (finished) created at:

github.com/filecoin-project/go-statemachine.(*StateMachine).run()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/go-statemachine@v0.0.0-20200925024713-05bd7c71fbfe/machine.go:100 +0x7ab


==================

WARNING: DATA RACE

Write at 0x00c000cd95d8 by goroutine 809:

github.com/filecoin-project/lotus/chain/events.(*tipSetCache).add()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/tscache.go:59 +0x359

github.com/filecoin-project/lotus/chain/events.(*heightEvents).headChangeAt()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events_height.go:84 +0x724

github.com/filecoin-project/lotus/chain/events.(*Events).headChange()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events.go:175 +0x20a

github.com/filecoin-project/lotus/chain/events.(*Events).listenHeadChangesOnce()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events.go:154 +0x7a4

github.com/filecoin-project/lotus/chain/events.(*Events).listenHeadChanges()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events.go:94 +0x11e


Previous read at 0x00c000cd95d8 by goroutine 347:

??()

-:0 +0xffffffffffffffff

github.com/filecoin-project/go-amt-ipld/v2.(*Root).UnmarshalCBOR()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/go-amt-ipld/v2@v2.1.1-0.20201006184820-924ee87a1349/cbor_gen.go:98 +0x354

github.com/ipfs/go-ipld-cbor.(*BasicIpldStore).Get()

/home/steb/.local/share/go/pkg/mod/github.com/ipfs/go-ipld-cbor@v0.0.5-0.20200428170625-a0bd04d3cbdf/store.go:51 +0x204

github.com/filecoin-project/specs-actors/actors/util/adt.(*wstore).Get()

:1 +0xb6

github.com/filecoin-project/go-amt-ipld/v2.LoadAMT()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/go-amt-ipld/v2@v2.1.1-0.20201006184820-924ee87a1349/amt.go:56 +0x101

github.com/filecoin-project/specs-actors/actors/util/adt.AsArray()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/specs-actors@v0.9.12/actors/util/adt/array.go:22 +0xe4

github.com/filecoin-project/lotus/chain/actors/builtin/market.(*state0).States()

/home/steb/src/github.com/filecoin-project/lotus/chain/actors/builtin/market/v0.go:61 +0xc7

github.com/filecoin-project/lotus/chain/stmgr.GetStorageDeal()

/home/steb/src/github.com/filecoin-project/lotus/chain/stmgr/utils.go:325 +0x566

github.com/filecoin-project/lotus/node/impl/full.(*StateAPI).StateMarketStorageDeal()

/home/steb/src/github.com/filecoin-project/lotus/node/impl/full/state.go:658 +0x237

github.com/filecoin-project/lotus/markets/storageadapter.(*ClientNodeAdapter).OnDealExpiredOrSlashed()

/home/steb/src/github.com/filecoin-project/lotus/markets/storageadapter/client.go:324 +0x257

github.com/filecoin-project/go-fil-markets/storagemarket/impl/clientstates.WaitForDealCompletion()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/go-fil-markets@v0.7.1/storagemarket/impl/clientstates/client_states.go:270 +0x193

runtime.call512()

/usr/lib/go/src/runtime/asm_amd64.s:544 +0x5b

reflect.Value.Call()

/usr/lib/go/src/reflect/value.go:336 +0xd8

github.com/filecoin-project/go-statemachine/fsm.fsmHandler.handler.func1()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/go-statemachine@v0.0.0-20200925024713-05bd7c71fbfe/fsm/fsm.go:170 +0x484

reflect.callReflect()

/usr/lib/go/src/reflect/value.go:564 +0x5c5

reflect.makeFuncStub()

/usr/lib/go/src/reflect/asm_amd64.s:20 +0x41

reflect.Value.Call()

/usr/lib/go/src/reflect/value.go:336 +0xd8

github.com/filecoin-project/go-statemachine.(*StateMachine).run.func3()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/go-statemachine@v0.0.0-20200925024713-05bd7c71fbfe/machine.go:102 +0x27c


Goroutine 809 (running) created at:

github.com/filecoin-project/lotus/chain/events.NewEvents()

/home/steb/src/github.com/filecoin-project/lotus/chain/events/events.go:83 +0x7c4

github.com/filecoin-project/lotus/markets/storageadapter.NewClientNodeAdapter()

/home/steb/src/github.com/filecoin-project/lotus/markets/storageadapter/client.go:61 +0x144

runtime.call512()

/usr/lib/go/src/runtime/asm_amd64.s:544 +0x5b

reflect.Value.Call()

/usr/lib/go/src/reflect/value.go:336 +0xd8

github.com/filecoin-project/lotus/node.as.func2()

/home/steb/src/github.com/filecoin-project/lotus/node/options.go:140 +0xb5

reflect.callReflect()

/usr/lib/go/src/reflect/value.go:564 +0x5c5

reflect.makeFuncStub()

/usr/lib/go/src/reflect/asm_amd64.s:20 +0x41

reflect.Value.Call()

/usr/lib/go/src/reflect/value.go:336 +0xd8

go.uber.org/dig.defaultInvoker()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:284 +0x7a

go.uber.org/dig.(*node).Call()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:710 +0x2d0

go.uber.org/dig.paramSingle.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x39a

go.uber.org/dig.(*paramSingle).Build()

:1 +0xe7

go.uber.org/dig.paramList.BuildList()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0xf7

go.uber.org/dig.(*node).Call()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0x155

go.uber.org/dig.paramSingle.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x39a

go.uber.org/dig.(*paramSingle).Build()

:1 +0xe7

go.uber.org/dig.paramObjectField.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:384 +0x1c3

go.uber.org/dig.paramObject.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:311 +0x160

go.uber.org/dig.(*paramObject).Build()

:1 +0xe7

go.uber.org/dig.paramObjectField.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:384 +0x1c3

go.uber.org/dig.paramObject.Build()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:311 +0x160

go.uber.org/dig.(*paramObject).Build()

:1 +0xe7

go.uber.org/dig.paramList.BuildList()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0xf7

go.uber.org/dig.(*Container).Invoke()

/home/steb/.local/share/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:432 +0x372

go.uber.org/fx.(*App).executeInvokes()

/home/steb/.local/share/go/pkg/mod/go.uber.org/fx@v1.9.0/app.go:547 +0x493

go.uber.org/fx.New()

/home/steb/.local/share/go/pkg/mod/go.uber.org/fx@v1.9.0/app.go:345 +0x83b

github.com/filecoin-project/lotus/node.New()

/home/steb/src/github.com/filecoin-project/lotus/node/builder.go:570 +0x64e

github.com/filecoin-project/lotus/node/test.mockBuilderOpts()

/home/steb/src/github.com/filecoin-project/lotus/node/test/builder.go:239 +0x18f2

github.com/filecoin-project/lotus/node/test.Builder()

/home/steb/src/github.com/filecoin-project/lotus/node/test/builder.go:145 +0xa8

github.com/filecoin-project/lotus/api/test.TestDealFlow()

/home/steb/src/github.com/filecoin-project/lotus/api/test/deals.go:51 +0x148

github.com/filecoin-project/lotus/node_test.TestAPIDealFlowReal.func3()

/home/steb/src/github.com/filecoin-project/lotus/node/node_test.go:78 +0x54

testing.tRunner()

/usr/lib/go/src/testing/testing.go:1127 +0x202


Goroutine 347 (finished) created at:

github.com/filecoin-project/go-statemachine.(*StateMachine).run()

/home/steb/.local/share/go/pkg/mod/github.com/filecoin-project/go-statemachine@v0.0.0-20200925024713-05bd7c71fbfe/machine.go:100 +0x7ab

Stebalien
Stebalien commented Oct 10, 2020
View reviewed changes
chain/events/tscache.go
@@ -95,14 +104,18 @@ func (tsc *tipSetCache) getNonNull(height abi.ChainEpoch) (*types.TipSet, error)
}

func (tsc *tipSetCache) get(height abi.ChainEpoch) (*types.TipSet, error) {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bit nasty but I'm trying to avoid calling into the storage subsystem while holding a lock.

@Stebalien
fix a race in tipset cache usage
f32b3bf 
This tipset cache is shared between multiple services and is called from
multiple places.
@magik6k magik6k merged commit 9a26896 into master Oct 10, 2020
@magik6k magik6k deleted the steb/fix-tipset-cache-race branch October 10, 2020 10:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@magik6k magik6k magik6k approved these changes

@Kubuxu Kubuxu Awaiting requested review from Kubuxu

@whyrusleeping whyrusleeping Awaiting requested review from whyrusleeping

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Stebalien @magik6k