Mainnet state migration crashes on preview.22 build

[j1010001]

Migration doc: https://www.notion.so/flowfoundation/MN-state-migration-test-3-Atree-inlining-C1-0-Apr-25-preview-14-5fdc4a3e89a64ed2ae2212a37bb950b6

Error:

2:58PM ERR failed to find all contract registers that need to be changed contracts={"0x631e88ae7f1d7c20":["NonFungibleToken","MetadataViews","ViewResolver"],"0x7aad92e5a0715d21":["StakingProxy"],"0x7e60df042a9c0868":["FlowToken"],"0x8c5303eaa26202d6":["FlowServiceAccount","NodeVersionBeacon","RandomBeaconHistory","FlowStorageFees"],"0x912d5440f7e3769e":["FlowFees"],"0x95e019a17d0e23d7":["FlowStakingCollection","LockedTokens"],"0x9a0766d93b6608b7":["FungibleTokenMetadataViews","FungibleTokenSwitchboard","Burner","FungibleToken"],"0x9eca2b38b18b5dfe":["FlowDKG","FlowEpoch","FlowIDTableStaking","FlowClusterQC"]} migration=SystemContractsMigration migration_index=0
2:58PM INF finished migrating Payloads account_count=31996628 payload_count=444847430
2:58PM INF migration 0 is done timeTaken=35m28.944298701s
2:58PM INF migration 2/7 is underway
2:58PM INF created account migrations inner_migrations=1 nWorker=160
2:58PM INF Sorting payloads by address payloads=444847430 workers=160
2:59PM INF Sorted. Finding account boundaries in sorted payloads duration=53s payloads=444847430
3:00PM INF Done grouping payloads by account accounts=31996628 duration=30s
3:04PM INF found 0 payloads in account 0x2ceae959ed1a7e7a migration=StagedContractsMigration migration_index=0
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1828b85]

goroutine 1 [running]:
github.com/onflow/flow-go/cmd/util/ledger/migrations.(*StagedContractsMigration).collectAndRegisterStagedContractsFromPayloads(0xc0008d8a00, {0xfa71b00000, 0x1a83d546, 0x0?})
        /app/cmd/util/ledger/migrations/staged_contracts_migration.go:243 +0x505
github.com/onflow/flow-go/cmd/util/ledger/migrations.(*StagedContractsMigration).InitMigration(0xc0008d8a00, {{0x2294390, 0xc00089e460}, 0xff, {0x0, 0x0}, {0xef7e073a00, 0x14, 0x1f4}, {0xc00089e470, ...}, ...}, ...)
        /app/cmd/util/ledger/migrations/staged_contracts_migration.go:135 +0x2ee
github.com/onflow/flow-go/cmd/util/ledger/migrations.MigrateByAccount({{0x2294390, 0xc00089e460}, 0xff, {0x0, 0x0}, {0xc000552e00, 0x1, 0x1f4}, {0xc00089e470, 0x1, ...}, ...}, ...)
        /app/cmd/util/ledger/migrations/account_based_migration.go:86 +0x43d
github.com/onflow/flow-go/cmd/util/ledger/migrations.NewAccountBasedMigration.func1({0xfa71b00000?, 0x1e79b62?, 0x0?})
        /app/cmd/util/ledger/migrations/account_based_migration.go:53 +0x6b
github.com/onflow/flow-go/cmd/util/cmd/execution-state-extract.migratePayloads({{0x2294390, 0xc00089e460}, 0xff, {0x0, 0x0}, {0xc000552e00, 0x1, 0x1f4}, {0xc00089e470, 0x1, ...}, ...}, ...)
        /app/cmd/util/cmd/execution-state-extract/execution_state_extract.go:323 +0x23b
github.com/onflow/flow-go/cmd/util/cmd/execution-state-extract.extractExecutionStateFromPayloads({{0x2294390, 0xc00089e460}, 0xff, {0x0, 0x0}, {0xc000552e00, 0x1, 0x1f4}, {0xc00089e470, 0x1, ...}, ...}, ...)
        /app/cmd/util/cmd/execution-state-extract/execution_state_extract.go:231 +0x1f9
github.com/onflow/flow-go/cmd/util/cmd/execution-state-extract.run(0x30c9400, {0x1e53053, 0xc, 0xc})
        /app/cmd/util/cmd/execution-state-extract/cmd.go:387 +0x1689
github.com/spf13/cobra.(*Command).execute(0x30c9400, {0xc00045f500, 0xc, 0xc})
        /go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:987 +0xa91
github.com/spf13/cobra.(*Command).ExecuteC(0x30c6bc0)
        /go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115 +0x425
github.com/spf13/cobra.(*Command).Execute(...)
        /go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039
github.com/onflow/flow-go/cmd/util/cmd.Execute()
        /app/cmd/util/cmd/root.go:50 +0x25
main.main()
        /app/cmd/util/main.go:8 +0x17

[SupunS]

Error is from: https://github.com/onflow/flow-go/blob/1c413fc5d2ecef8c6e132cd547954135b97fba03/cmd/util/ledger/migrations/staged_contracts_migration.go#L242-L243

Looks like the staging account storage for MN is not found? The address is correct though: https://contractbrowser.com/account/0x56100d46aa9b0212

[turbolent]

Error is from: onflow/flow-go@1c413fc/cmd/util/ledger/migrations/staged_contracts_migration.go#L242-L243

Looks like the staging account storage for MN is not found? The address is correct though: contractbrowser.com/account/0x56100d46aa9b0212

Yeah, we should maybe check if storageMap is nil and report an error.

https://f.dnz.dev/56100d46aa9b0212 shows there are staged contracts in storage.

@sisyphusSmiling Can you confirm this is the correct account on MN?

[SupunS]

oh hang on:

3:04PM INF found 0 payloads in account 0x2ceae959ed1a7e7a migration=StagedContractsMigration migration_index=0

The migration has picked the wrong address 0x2ceae959ed1a7e7a (it's for the testnet)

[SupunS]

@j1010001 can you maybe double-check/re-run the command with --chain flow-mainnet

[turbolent]

The migration has picked the wrong address 0x2ceae959ed1a7e7a (it's for the testnet)

Oh, good catch!

@onflow/flow-cadence-execution Is there any way to check that the chain ID flag matches the chain ID of the state, or even better, can we maybe just remove the chain ID flag and use the state's flag?

The flag has been brittle, mixups like this have happened before.

[j1010001]

I have noticed I also had the --chain flow-testnet flag in the atree register inlining migration - @fxamacker is it used there ? I am guessing it isn't as the migration completed successfully, but let me know if you think it makes the migrated state invalid for some reason and I can re-run both.

[fxamacker]

I have noticed I also had the --chain flow-testnet flag in the atree register inlining migration - @fxamacker is it used there ? I am guessing it isn't as the migration completed successfully, but let me know if you think it makes the migrated state invalid for some reason and I can re-run both.

@j1010001 Yes, --chain is used by flag --fix-testnet-slabs-with-broken-references in atree inlining migration. Out of abundance of caution and to reduce debugging time, I think we should re-run both migration programs.

[j1010001]

Closing the issue - the crash was caused by using wrong --chain parameter on Mainnet state migration.

[janezpodhostnik]

@turbolent @j1010001

I was looking into removing the chainID parameter and it looks like that might complicate things and make the migrations longer (because it wouldn't be available without reading the service account from the state).

However I think we could still add a check somewhere deeper in (perhaps into the first migration) that the addresses are valid for the chainID provided. This way we would fail faster in the case of specifying the incorrect chainID.

[turbolent]

@janezpodhostnik Thank you for looking into it!

Yeah, we don't necessarily have to remove it, checking if it is correct as early as possible is also great. Would you be able to add that?

[janezpodhostnik]

Yes I can add that. I'll connect it to this issue.

[turbolent]

@janezpodhostnik Any update here? (No rush)

[janezpodhostnik]

Sorry, not yet. This is currently 2nd I'm my priority list.

[j1010001]

Closing, I opened #3335 for the remaining task