Restart after automatic upgrade not working ( Module Add-Delete-Add )

[vishnukumavat]

Summary of Bug

After an automatic upgrade, if you stop the chain at height X and re-start it the following error is returned:

failed to load latest version: failed to load store: initial version set to X (upgradeHeight +1), but found earlier version 1

Version

Cosmos SDK v0.45.4

Steps to Reproduce

I discovered this while exploring automatic upgrades and how it works in Cosmos SDK,
and also to test if the upgrade works in particular situations/conditions.

Note: Please correct me if I'm missing anything here, I'm just exploring stuff and learning.

Previous Issue #8265

1. Initialize the dummy (foo) module in the chain.

   • action - Chain started
   • result - success
   • reason - initialized a new module, assuming this is what needed to be on-chain with some basic features and states

2. Raise a software upgrade proposal; this proposal includes deletion of the (foo) module from the chain.

   • action - Chain restarted with upgraded binary, and without foo module

   • result - success

   • reason -
     a. This module wasn't properly written.
     b. Removal of all the states which are useless (or not needed anymore)
     c. The module has never been used
     d. The features provided by modules are not great.

   • eg.

storeUpgrades = &storetypes.StoreUpgrades{
	Deleted: []string{"foo"}
}
app.SetStoreLoader(upgradetypes.UpgradeStoreLoader(upgradeInfo.Height, storeUpgrades))

3. Raise a software upgrade proposal; this proposal includes re-adding (foo) module on-chain.
   • action - Rebuild the binary, and start the chain
   • result - error -failed to load latest version: failed to load store: initial version set to 40, but found earlier version 1
   • reason -
     a. Built from scratch again.
     b. New states, msgs, and queries.
     c. New features added.
   • eg.

storeUpgrades = &storetypes.StoreUpgrades{
	Added: []string{"foo"}
}
app.SetStoreLoader(upgradetypes.UpgradeStoreLoader(upgradeInfo.Height, storeUpgrades))

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[lucaslopezf]

Hi @vishnukumavat, the problem arises due to a version conflict in the database. Here is a detailed explanation of the scenario:

Scenario

1. Initial Addition: The module counter is added to the blockchain at block 1.
2. Deletion: The module counter is deleted at block 25. This means the module was last seen in the blockchain at block 24.
3. Re-Addition: We attempt to re-add the module counter at block 50. According to the upgrade proposal, the module should be seen for the first time at block 50. However, the database still retains the information that the module was first seen at block 1.

The initial version is retrieved from the database here:
iavl/nodedb.go#L709

The function where the version is retrieved:
iavl/mutable_tree.go#L441

This version conflict occurs because the SDK's internal state management does not account for the possibility of a module being removed and then reintroduced with the same identifier. The database retains historical data about the module's initial addition, which leads to inconsistencies when the module is re-added. Specifically, the SDK's state expects the module to be new at block 50, but the database's historical records show that the module was first added at block 1. This discrepancy causes the system to fail when trying to reconcile the module's state across different blocks.

Suggested Solution

Ideally, if a module needs to be updated, it should be done through an update process rather than deletion and re-addition. However, if a module has already been deleted and needs to be re-added, it should be added with a different name, such as appending a version number (e.g., foo_v2). This will ensure that the new module is treated as a completely new entity in the database, avoiding any version conflicts.

Also, I am including debugging images that may help to understand the problem.

[educlerici-zondax]

@vishnukumavat were you able to check lucas's answer?