-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
upgrade-info.json
not dumped correctly
#10428
Comments
upgrade-info.json
not dumped correctly
UpdateYesterday we had another on-chain upgrade always using Cosmovisor 1.0.0. The same thing happened: the upgrade plan was not dumped correctly and the chain halted. Upgrade proposal (one block before upgrade height)
On-chain upgrade plan (one block before the upgrade height)
Dumped JSON (at the upgrade height)
Error logs
|
Thanks for a clear and detailed report. It looks to be urgent. |
@yaruwangway - do you have time to investigate this issue? |
Hi @RiccardoM , in this
|
Hi @RiccardoM , can you also let me know from which sdk version you are upgrading to sdk version v0.44.2 ? |
@yaruwangway We can't try editing the JSON file cause the upgrade plan contains a checksum of it and changing the contents would cause the hashsum to be invalid and the upgrade to fail. Plus, we managed to solve the issue and the upgrade has now passed. I would refrain from testing this on our running testnet since it might cause other disruptions. It might be better to have some unit test setup instead either inside Cosmovisor or the About your second question, we were not upgrading to any Cosmos SDK version during neither of the two mentioned upgrades. The Cosmos SDK upgrade was performed with a previous upgrade that did not have any problem. |
@RiccardoM , did you identify the issue? |
@robert-zaremba No I haven't. I've checked the |
After doing some research on this, there's a couple things I'd like to address. Automatic upgrades not working:The upgrade module in Basically, the most recent version of Error message in
|
## Description This PR updates Cosmos to `v0.44.3` and also cherry picks the changes that have been made inside cosmos/cosmos-sdk#8590 to fix the issue described in cosmos/cosmos-sdk#10428 <!-- Add a description of the changes that this PR introduces and the files that are the most critical to review. --> --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] added `!` to the type prefix if API or client breaking change - [x] targeted the correct branch (see [PR Targeting](https://github.com/desmos-labs/desmos/blob/master/CONTRIBUTING.md#pr-targeting)) - [ ] provided a link to the relevant issue or specification - [ ] followed the guidelines for [building modules](https://docs.cosmos.network/v0.44/building-modules/intro.html) - [ ] included the necessary unit and integration [tests](https://github.com/desmos-labs/desmos/blob/master/CONTRIBUTING.md#testing) - [x] added a changelog entry to `CHANGELOG.md` - [ ] included comments for [documenting Go code](https://blog.golang.org/godoc) - [ ] updated the relevant documentation or specification - [x] reviewed "Files changed" and left comments if necessary - [x] confirmed all CI checks have passed ### Reviewers Checklist *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable)
Indeed we were missing Info in the dumped file. This is in master but was not there in |
Summary of bug
During an on-chain upgrade, the
upgrade-info.json
file is not dumped correctly, resulting in a chain halt.Version
v0.44.2
Steps to reproduce
Context
Inside our testnet, we have always used Cosmovisor v.0.1.0 in order to manage the various on-chain upgrades that we had. It always worked fine: the upgrade info was read properly from the chain and the binaries were downloaded and run correctly.
Recently, since v.0.1.0 does not support automatic downloads for Cosmos v0.44.x binaries (see here), we asked every node operator to update to Cosmovisor v1.0.0. Unfortunately, it caused a lot of errors during various phases. Following, the whole things we did when upgrading to Cosmovisor v1.0.0 and all the errors that were raised.
Upgrading procedure
1. We installed the correct Cosmovisor version running
2. We tried verifying the running Cosmovisor version:
This raised the first error:
This seems caused by the fact that Cosmovisor searches for that file and it raises an error when not found. However, after running
cosmovisor start
the chain re-starts properly without much problems even if that file is not present.3. We tried an on-chain upgrade. This is were things went really bad, even halting our chain.
We submitted the following upgrade proposal on our testnet the same we always did: Desmos v2.1.0 upgrade proposal. We specified an upgrade info link that points to a valid JSON file.
When the upgrade height came, however, the upgrade failed with the following error:
I then took a look at the
~/.desmos/data/upgrade-info.json
file and it looked like this:It appears that the upgrade info is not dumped properly, even thought it was present on chain (see the proposal details here). This prevented Cosmovisor from downloading the binaries, effectively halting our chain.
We solve this problem by manually replacing the
~/.desmos/data/upgrade-info.json
contents adding theinfo
field:After restarting Cosmovisor, this triggered the upgrade successfully and started downloading the files. It even copied the
upgrade-info.json
contents correctly inside the~/.desmos/cosmovisor/upgrades/v2.1.0/upgrade-info.json
file.Additional info
For Admin Use
The text was updated successfully, but these errors were encountered: