Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Payload grouping has data races #5485

Closed
1 task done
Tracked by #3096
turbolent opened this issue Feb 29, 2024 · 0 comments · Fixed by #5488
Closed
1 task done
Tracked by #3096

Payload grouping has data races #5485

turbolent opened this issue Feb 29, 2024 · 0 comments · Fixed by #5488
Labels
Bug Something isn't working

Comments

@turbolent
Copy link
Member

turbolent commented Feb 29, 2024 •
edited
Loading

🐞 Bug Report

GroupPayloadsByAccount has data races.

What is the severity of this bug?

Critical: We can't do anything if this isn't actioned immediately (product doesn't function without this, it's blocking us or users, or it resolves a high severity security issue). One person should look at this right now.

Reproduction steps

Steps to reproduce the behaviour:

  1. Build util binary with race detector enabled
  2. Run account-based migration with worker count >1
  • I followed these steps and was able to reproduce the issue

Expected behaviour

The race detector should not report any errors.

Screenshots

Report 
==================
WARNING: DATA RACE
Write at 0x00c31072dd50 by goroutine 212:
  runtime.slicecopy()
      /opt/homebrew/Cellar/go@1.20/1.20.12/libexec/src/runtime/slice.go:310 +0x0
  github.com/onflow/flow-go/cmd/util/ledger/util.mergeInto()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/util/payload_grouping.go:252 +0x1a9
  github.com/onflow/flow-go/cmd/util/ledger/util.sortPayloads()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/util/payload_grouping.go:240 +0x525
  github.com/onflow/flow-go/cmd/util/ledger/util.sortPayloads.func1()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/util/payload_grouping.go:231 +0xcd

Previous write at 0x00c31072dd50 by goroutine 213:
  runtime.slicecopy()
      /opt/homebrew/Cellar/go@1.20/1.20.12/libexec/src/runtime/slice.go:310 +0x0
  github.com/onflow/flow-go/cmd/util/ledger/util.mergeInto()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/util/payload_grouping.go:252 +0x1a9
  github.com/onflow/flow-go/cmd/util/ledger/util.sortPayloads()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/util/payload_grouping.go:240 +0x525
  github.com/onflow/flow-go/cmd/util/ledger/util.sortPayloads.func2()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/util/payload_grouping.go:235 +0xcd

Goroutine 212 (running) created at:
  github.com/onflow/flow-go/cmd/util/ledger/util.sortPayloads()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/util/payload_grouping.go:230 +0x34a
  github.com/onflow/flow-go/cmd/util/ledger/util.GroupPayloadsByAccount()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/util/payload_grouping.go:102 +0x12e
  github.com/onflow/flow-go/cmd/util/ledger/migrations.MigrateByAccount()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/migrations/account_based_migration.go:82 +0x124
  github.com/onflow/flow-go/cmd/util/ledger/migrations.NewAccountBasedMigration.func1()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/migrations/account_based_migration.go:52 +0x10f
  github.com/onflow/flow-go/cmd/util/cmd/execution-state-extract.migratePayloads()
      /Users/bastian/Documents/work/flow-go/cmd/util/cmd/execution-state-extract/execution_state_extract.go:320 +0x28c
  github.com/onflow/flow-go/cmd/util/cmd/execution-state-extract.extractExecutionStateFromPayloads()
      /Users/bastian/Documents/work/flow-go/cmd/util/cmd/execution-state-extract/execution_state_extract.go:232 +0x21b
  github.com/onflow/flow-go/cmd/util/cmd/execution-state-extract.run()
      /Users/bastian/Documents/work/flow-go/cmd/util/cmd/execution-state-extract/cmd.go:296 +0x1384
  github.com/spf13/cobra.(*Command).execute()
      /Users/bastian/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:987 +0xf01
  github.com/spf13/cobra.(*Command).ExecuteC()
      /Users/bastian/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115 +0x63a
  github.com/spf13/cobra.(*Command).Execute()
      /Users/bastian/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039 +0x3c
  github.com/onflow/flow-go/cmd/util/cmd.Execute()
      /Users/bastian/Documents/work/flow-go/cmd/util/cmd/root.go:47 +0x25
  main.main()
      /Users/bastian/Documents/work/flow-go/cmd/util/main.go:8 +0x24

Goroutine 213 (running) created at:
  github.com/onflow/flow-go/cmd/util/ledger/util.sortPayloads()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/util/payload_grouping.go:234 +0x4ce
  github.com/onflow/flow-go/cmd/util/ledger/util.GroupPayloadsByAccount()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/util/payload_grouping.go:102 +0x12e
  github.com/onflow/flow-go/cmd/util/ledger/migrations.MigrateByAccount()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/migrations/account_based_migration.go:82 +0x124
  github.com/onflow/flow-go/cmd/util/ledger/migrations.NewAccountBasedMigration.func1()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/migrations/account_based_migration.go:52 +0x10f
  github.com/onflow/flow-go/cmd/util/cmd/execution-state-extract.migratePayloads()
      /Users/bastian/Documents/work/flow-go/cmd/util/cmd/execution-state-extract/execution_state_extract.go:320 +0x28c
  github.com/onflow/flow-go/cmd/util/cmd/execution-state-extract.extractExecutionStateFromPayloads()
      /Users/bastian/Documents/work/flow-go/cmd/util/cmd/execution-state-extract/execution_state_extract.go:232 +0x21b
  github.com/onflow/flow-go/cmd/util/cmd/execution-state-extract.run()
      /Users/bastian/Documents/work/flow-go/cmd/util/cmd/execution-state-extract/cmd.go:296 +0x1384
  github.com/spf13/cobra.(*Command).execute()
      /Users/bastian/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:987 +0xf01
  github.com/spf13/cobra.(*Command).ExecuteC()
      /Users/bastian/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115 +0x63a
  github.com/spf13/cobra.(*Command).Execute()
      /Users/bastian/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039 +0x3c
  github.com/onflow/flow-go/cmd/util/cmd.Execute()
      /Users/bastian/Documents/work/flow-go/cmd/util/cmd/root.go:47 +0x25
  main.main()
      /Users/bastian/Documents/work/flow-go/cmd/util/main.go:8 +0x24
==================
==================
WARNING: DATA RACE
Write at 0x00c00ccc2108 by goroutine 213:
  runtime.slicecopy()
      /opt/homebrew/Cellar/go@1.20/1.20.12/libexec/src/runtime/slice.go:310 +0x0
  github.com/onflow/flow-go/cmd/util/ledger/util.mergeInto()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/util/payload_grouping.go:274 +0x3ef
  github.com/onflow/flow-go/cmd/util/ledger/util.sortPayloads()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/util/payload_grouping.go:240 +0x525
  github.com/onflow/flow-go/cmd/util/ledger/util.sortPayloads.func2()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/util/payload_grouping.go:235 +0xcd

Previous read at 0x00c00ccc2108 by goroutine 212:
  runtime.slicecopy()
      /opt/homebrew/Cellar/go@1.20/1.20.12/libexec/src/runtime/slice.go:310 +0x0
  github.com/onflow/flow-go/cmd/util/ledger/util.mergeInto()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/util/payload_grouping.go:252 +0x1a9
  github.com/onflow/flow-go/cmd/util/ledger/util.sortPayloads()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/util/payload_grouping.go:240 +0x525
  github.com/onflow/flow-go/cmd/util/ledger/util.sortPayloads.func1()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/util/payload_grouping.go:231 +0xcd

Goroutine 213 (running) created at:
  github.com/onflow/flow-go/cmd/util/ledger/util.sortPayloads()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/util/payload_grouping.go:234 +0x4ce
  github.com/onflow/flow-go/cmd/util/ledger/util.GroupPayloadsByAccount()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/util/payload_grouping.go:102 +0x12e
  github.com/onflow/flow-go/cmd/util/ledger/migrations.MigrateByAccount()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/migrations/account_based_migration.go:82 +0x124
  github.com/onflow/flow-go/cmd/util/ledger/migrations.NewAccountBasedMigration.func1()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/migrations/account_based_migration.go:52 +0x10f
  github.com/onflow/flow-go/cmd/util/cmd/execution-state-extract.migratePayloads()
      /Users/bastian/Documents/work/flow-go/cmd/util/cmd/execution-state-extract/execution_state_extract.go:320 +0x28c
  github.com/onflow/flow-go/cmd/util/cmd/execution-state-extract.extractExecutionStateFromPayloads()
      /Users/bastian/Documents/work/flow-go/cmd/util/cmd/execution-state-extract/execution_state_extract.go:232 +0x21b
  github.com/onflow/flow-go/cmd/util/cmd/execution-state-extract.run()
      /Users/bastian/Documents/work/flow-go/cmd/util/cmd/execution-state-extract/cmd.go:296 +0x1384
  github.com/spf13/cobra.(*Command).execute()
      /Users/bastian/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:987 +0xf01
  github.com/spf13/cobra.(*Command).ExecuteC()
      /Users/bastian/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115 +0x63a
  github.com/spf13/cobra.(*Command).Execute()
      /Users/bastian/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039 +0x3c
  github.com/onflow/flow-go/cmd/util/cmd.Execute()
      /Users/bastian/Documents/work/flow-go/cmd/util/cmd/root.go:47 +0x25
  main.main()
      /Users/bastian/Documents/work/flow-go/cmd/util/main.go:8 +0x24

Goroutine 212 (running) created at:
  github.com/onflow/flow-go/cmd/util/ledger/util.sortPayloads()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/util/payload_grouping.go:230 +0x34a
  github.com/onflow/flow-go/cmd/util/ledger/util.GroupPayloadsByAccount()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/util/payload_grouping.go:102 +0x12e
  github.com/onflow/flow-go/cmd/util/ledger/migrations.MigrateByAccount()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/migrations/account_based_migration.go:82 +0x124
  github.com/onflow/flow-go/cmd/util/ledger/migrations.NewAccountBasedMigration.func1()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/migrations/account_based_migration.go:52 +0x10f
  github.com/onflow/flow-go/cmd/util/cmd/execution-state-extract.migratePayloads()
      /Users/bastian/Documents/work/flow-go/cmd/util/cmd/execution-state-extract/execution_state_extract.go:320 +0x28c
  github.com/onflow/flow-go/cmd/util/cmd/execution-state-extract.extractExecutionStateFromPayloads()
      /Users/bastian/Documents/work/flow-go/cmd/util/cmd/execution-state-extract/execution_state_extract.go:232 +0x21b
  github.com/onflow/flow-go/cmd/util/cmd/execution-state-extract.run()
      /Users/bastian/Documents/work/flow-go/cmd/util/cmd/execution-state-extract/cmd.go:296 +0x1384
  github.com/spf13/cobra.(*Command).execute()
      /Users/bastian/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:987 +0xf01
  github.com/spf13/cobra.(*Command).ExecuteC()
      /Users/bastian/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115 +0x63a
  github.com/spf13/cobra.(*Command).Execute()
      /Users/bastian/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039 +0x3c
  github.com/onflow/flow-go/cmd/util/cmd.Execute()
      /Users/bastian/Documents/work/flow-go/cmd/util/cmd/root.go:47 +0x25
  main.main()
      /Users/bastian/Documents/work/flow-go/cmd/util/main.go:8 +0x24
==================
==================
WARNING: DATA RACE
Write at 0x00c313590328 by goroutine 212:
  runtime.slicecopy()
      /opt/homebrew/Cellar/go@1.20/1.20.12/libexec/src/runtime/slice.go:310 +0x0
  github.com/onflow/flow-go/cmd/util/ledger/util.mergeInto()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/util/payload_grouping.go:271 +0x37a
  github.com/onflow/flow-go/cmd/util/ledger/util.sortPayloads()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/util/payload_grouping.go:240 +0x525
  github.com/onflow/flow-go/cmd/util/ledger/util.sortPayloads.func1()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/util/payload_grouping.go:231 +0xcd

Previous write at 0x00c313590328 by goroutine 213:
  runtime.slicecopy()
      /opt/homebrew/Cellar/go@1.20/1.20.12/libexec/src/runtime/slice.go:310 +0x0
  github.com/onflow/flow-go/cmd/util/ledger/util.mergeInto()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/util/payload_grouping.go:252 +0x1a9
  github.com/onflow/flow-go/cmd/util/ledger/util.sortPayloads()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/util/payload_grouping.go:240 +0x525
  github.com/onflow/flow-go/cmd/util/ledger/util.sortPayloads.func2()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/util/payload_grouping.go:235 +0xcd

Goroutine 212 (running) created at:
  github.com/onflow/flow-go/cmd/util/ledger/util.sortPayloads()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/util/payload_grouping.go:230 +0x34a
  github.com/onflow/flow-go/cmd/util/ledger/util.GroupPayloadsByAccount()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/util/payload_grouping.go:102 +0x12e
  github.com/onflow/flow-go/cmd/util/ledger/migrations.MigrateByAccount()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/migrations/account_based_migration.go:82 +0x124
  github.com/onflow/flow-go/cmd/util/ledger/migrations.NewAccountBasedMigration.func1()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/migrations/account_based_migration.go:52 +0x10f
  github.com/onflow/flow-go/cmd/util/cmd/execution-state-extract.migratePayloads()
      /Users/bastian/Documents/work/flow-go/cmd/util/cmd/execution-state-extract/execution_state_extract.go:320 +0x28c
  github.com/onflow/flow-go/cmd/util/cmd/execution-state-extract.extractExecutionStateFromPayloads()
      /Users/bastian/Documents/work/flow-go/cmd/util/cmd/execution-state-extract/execution_state_extract.go:232 +0x21b
  github.com/onflow/flow-go/cmd/util/cmd/execution-state-extract.run()
      /Users/bastian/Documents/work/flow-go/cmd/util/cmd/execution-state-extract/cmd.go:296 +0x1384
  github.com/spf13/cobra.(*Command).execute()
      /Users/bastian/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:987 +0xf01
  github.com/spf13/cobra.(*Command).ExecuteC()
      /Users/bastian/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115 +0x63a
  github.com/spf13/cobra.(*Command).Execute()
      /Users/bastian/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039 +0x3c
  github.com/onflow/flow-go/cmd/util/cmd.Execute()
      /Users/bastian/Documents/work/flow-go/cmd/util/cmd/root.go:47 +0x25
  main.main()
      /Users/bastian/Documents/work/flow-go/cmd/util/main.go:8 +0x24

Goroutine 213 (finished) created at:
  github.com/onflow/flow-go/cmd/util/ledger/util.sortPayloads()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/util/payload_grouping.go:234 +0x4ce
  github.com/onflow/flow-go/cmd/util/ledger/util.GroupPayloadsByAccount()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/util/payload_grouping.go:102 +0x12e
  github.com/onflow/flow-go/cmd/util/ledger/migrations.MigrateByAccount()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/migrations/account_based_migration.go:82 +0x124
  github.com/onflow/flow-go/cmd/util/ledger/migrations.NewAccountBasedMigration.func1()
      /Users/bastian/Documents/work/flow-go/cmd/util/ledger/migrations/account_based_migration.go:52 +0x10f
  github.com/onflow/flow-go/cmd/util/cmd/execution-state-extract.migratePayloads()
      /Users/bastian/Documents/work/flow-go/cmd/util/cmd/execution-state-extract/execution_state_extract.go:320 +0x28c
  github.com/onflow/flow-go/cmd/util/cmd/execution-state-extract.extractExecutionStateFromPayloads()
      /Users/bastian/Documents/work/flow-go/cmd/util/cmd/execution-state-extract/execution_state_extract.go:232 +0x21b
  github.com/onflow/flow-go/cmd/util/cmd/execution-state-extract.run()
      /Users/bastian/Documents/work/flow-go/cmd/util/cmd/execution-state-extract/cmd.go:296 +0x1384
  github.com/spf13/cobra.(*Command).execute()
      /Users/bastian/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:987 +0xf01
  github.com/spf13/cobra.(*Command).ExecuteC()
      /Users/bastian/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115 +0x63a
  github.com/spf13/cobra.(*Command).Execute()
      /Users/bastian/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039 +0x3c
  github.com/onflow/flow-go/cmd/util/cmd.Execute()
      /Users/bastian/Documents/work/flow-go/cmd/util/cmd/root.go:47 +0x25
  main.main()
      /Users/bastian/Documents/work/flow-go/cmd/util/main.go:8 +0x24
==================

Specifications

  • System: Linux
  • Go: 1.20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix data race in account grouping for account based migration onflow/flow-go
2 participants
@turbolent @j1010001