exp/lighthorizon: mapreduce data race condition

[2opremio]

The tests at https://github.com/stellar/go/runs/7388078175?check_suite_focus=true detected a race condition

     mapreduce_test.go:45: Building baseline for ledger range [1410048, 1410303]
    mapreduce_test.go:202: 
        	Error Trace:	mapreduce_test.go:202
        	            				mapreduce_test.go:57
        	Error:      	map[uint32]struct {}{} does not contain 0x5614
        	Test:       	TestReduce
        	Messages:   	failed to find 22036 for GBM3LXCLBTCWHWHUZK3UCAVLE5SK3RVPIUIZIWW5V62FLTV4D6NHINCS (found map[])
==================
WARNING: DATA RACE
Write at 0x00c0006ce150 by goroutine 24:
  runtime.mapassign_faststr()
      /opt/hostedtoolcache/go/1.17.11/x64/src/runtime/map_faststr.go:202 +0x0
  github.com/stellar/go/exp/lighthorizon/index.(*store).getCreateIndex()
      /home/runner/work/go/go/exp/lighthorizon/index/store.go:313 +0x5a4
  github.com/stellar/go/exp/lighthorizon/index.(*store).AddParticipantsToIndexes()
      /home/runner/work/go/go/exp/lighthorizon/index/store.go:260 +0x184
  github.com/stellar/go/exp/lighthorizon/index.ProcessAccounts()
      /home/runner/work/go/go/exp/lighthorizon/index/modules.go:33 +0x202
  github.com/stellar/go/exp/lighthorizon/index.(*IndexBuilder).RunModules()
      /home/runner/work/go/go/exp/lighthorizon/index/builder.go:206 +0x152
  github.com/stellar/go/exp/lighthorizon/index.(*IndexBuilder).Build()
      /home/runner/work/go/go/exp/lighthorizon/index/builder.go:244 +0x62e
  github.com/stellar/go/exp/lighthorizon/index.BuildIndices.func2()
      /home/runner/work/go/go/exp/lighthorizon/index/builder.go:123 +0x1e4
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57 +0x96
Previous read at 0x00c0006ce150 by goroutine 32:
  runtime.mapiternext()
      /opt/hostedtoolcache/go/1.17.11/x64/src/runtime/map.go:851 +0x0
  github.com/stellar/go/exp/lighthorizon/index/backend.parallelFlush.func1()
      /home/runner/work/go/go/exp/lighthorizon/index/backend/parallel_flush.go:25 +0xa8
Goroutine 24 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:54 +0xf0
  github.com/stellar/go/exp/lighthorizon/index.BuildIndices()
      /home/runner/work/go/go/exp/lighthorizon/index/builder.go:117 +0xef7
  github.com/stellar/go/exp/lighthorizon/index/cmd_test.testSingleProcess()
      /home/runner/work/go/go/exp/lighthorizon/index/cmd/single_test.go:84 +0x274
  github.com/stellar/go/exp/lighthorizon/index/cmd_test.TestSingleProcess.func1()
      /home/runner/work/go/go/exp/lighthorizon/index/cmd/single_test.go:55 +0x46
  testing.tRunner()
      /opt/hostedtoolcache/go/1.17.11/x64/src/testing/testing.go:1259 +0x22f
  testing.(*T).Run·dwrap·21()
      /opt/hostedtoolcache/go/1.17.11/x64/src/testing/testing.go:1306 +0x47
Goroutine 32 (finished) created at:
  github.com/stellar/go/exp/lighthorizon/index/backend.parallelFlush()
      /home/runner/work/go/go/exp/lighthorizon/index/backend/parallel_flush.go:24 +0x13c
  github.com/stellar/go/exp/lighthorizon/index/backend.(*FileBackend).Flush()
      /home/runner/work/go/go/exp/lighthorizon/index/backend/file.go:30 +0xae
  github.com/stellar/go/exp/lighthorizon/index.(*store).Flush()
      /home/runner/work/go/go/exp/lighthorizon/index/store.go:165 +0x158
  github.com/stellar/go/exp/lighthorizon/index.BuildIndices.func2()
      /home/runner/work/go/go/exp/lighthorizon/index/builder.go:135 +0x259
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57 +0x96
==================
--- FAIL: TestSingleProcess (73.66s)
    single_test.go:76: Validating single-process builder on ledger range [1410048, 1410367] (320 ledgers)
    single_test.go:81: Storing indices in file:/tmp/TestSingleProcess1447992593/001
    --- FAIL: TestSingleProcess/start-at-checkpoint (37.55s)
        testing.go:1152: race detected during execution of test
    single_test.go:76: Validating single-process builder on ledger range [1410053, 1410367] (315 ledgers)
    single_test.go:81: Storing indices in file:/tmp/TestSingleProcess1447992593/002
    testing.go:1152: race detected during execution of test
FAIL

[Shaptic]

Yep, and it's not even a race condition 😕

This got introduced by #4442 somehow... I'm investigating.

[Shaptic]

This should be closed by #4465.

Nope, still present, though that did fix a real test failure. Still investigating...

[Shaptic]

Closed by #4470.