How to enable IBC on a live chain?

[Srutip04]

Summary

Hello,

We are currently running a live chain based on the Cosmos SDK, and we are interested in enabling Inter-Blockchain Communication (IBC) on our network. Since IBC was not initially enabled, we would like to know the proper steps to activate IBC on our live chain without causing disruptions.

Specific Questions:

1. Does enabling IBC require code-level changes to our existing application?
   If yes, what specific changes are necessary? Could you please outline the key modifications needed in the codebase, such as adding IBC modules (ibc, transfer, etc.), updating app.go, or modifying the genesis file?

2. What is the recommended procedure for rolling out IBC on a live chain?
   Could you provide a step-by-step guide for upgrading a live network to support IBC? We would like to understand how to safely transition from our current version to one that includes IBC, considering:

   • Preparing the new version with IBC enabled.
   • Coordinating with validators for the upgrade.
   • Proposing and implementing a software upgrade through governance.

3. Are there any best practices or potential pitfalls to be aware of during this process?
   Based on your experience, are there any common issues or challenges that we should anticipate when enabling IBC on a live chain?

4. What testing strategies do you recommend post-upgrade?
   Once IBC is enabled, what testing procedures should we follow to ensure everything is functioning correctly? Are there specific tools or methods we should use to verify IBC functionality, such as testing with a relayer like hermes?

We appreciate your guidance on this matter, as we aim to implement IBC with minimal risk to our network’s stability.

Thank you for your support!

[gjermundgaraba]

Hi @Srutip04!

I just wanted to give you an initial answer that hopefully will help you get started, and perhaps someone can help you on some of the other details. I will try to answer 1 and 2, in particular.

1. Does enabling IBC require code-level changes to our existing application?

Yes. You will need to integrate the IBC module from ibc-go (http://github.com/cosmos/ibc-go), the light clients you want to use (at least 07-tendermint, I would imagine, to enable Cosmos SDK chains to connect) and the applications you want to support (transfer, maybe interchain accounts). There are also some middleware modules that exist if you need packet forwarding for instance (https://github.com/cosmos/ibc-apps).

The exact version of IBC you can use depends on the version of the Cosmos SDK you are on. For instance, if you are on SDK 0.50, then the newest version of IBC, 8.4.0 would be the way to go. If you are on 0.47, then you should target 7.5.0.

For the integration part itself, I recommend taking a look at the IBC documentation that outlines the code changes needed: https://ibc.cosmos.network/v8/ibc/integration/
Another useful resource I have found is to look at how other chains have integrated the particular version of IBC you are going to use (because the exact integration might differ slightly between versions). An easy starting point would be for instance looking at the simapp from the ibc-go repo at the release tag of your selected IBC version (e.g. 8.4.x: https://github.com/cosmos/ibc-go/tree/v8.4.0/testing/simapp).

For the upgrade code itself, this is more Cosmos SDK specific and the main part you would have to do is creating an upgrade handler (https://docs.cosmos.network/v0.50/learn/advanced/upgrade#upgrade-handlers, adjust your version of the SDK for correct documention) and adding the modules during the upgrade (https://docs.cosmos.network/v0.50/learn/advanced/upgrade#adding-new-modules-during-upgrades). There are plenty of examples of chains who are adding modules during upgrades (although, I am not aware of any in particular that has added IBC after genesis myself). For instance, the Cosmos Hub added cosmwasm in their v18 upgrade, which you can see here: https://github.com/cosmos/gaia/tree/v18.0.0/app/upgrades/v18
If you need more examples, I can provide those as well.

You should not need to modify your genesis file, as the Cosmos SDK have excellent upgrade management.

2. What is the recommended procedure for rolling out IBC on a live chain?

The main procedure I recommend and have done myself in the past is the following high-level flow:

1. Implement the changes mentioned in the first answer
2. Write tests automated tests for your IBC use cases (e.g. transfer, ICA, etc) using interchaintest (http://github.com/strangelove-ventures/interchaintest)
3. Ideally also write an upgrade test suite (with interchaintest) that spins up the current version of the chain from a docker image, performs the upgrade using governance and finally runs some IBC smoke tests. There are some examples of this in the wild, for instance https://github.com/Neutaro/Neutaro/blob/main/interchaintest/upgrade_test.go (I am sure there are other good examples, but I happen to know this one since I wrote it :)). There are also some importable e2e test flows from the IBC repo you can look at: https://github.com/cosmos/ibc-go/tree/main/e2e#importable-workflow
4. Tag a pre-release version of your chain (good for the next step, so all validators on testnet have the same version) and make sure to have instructions on how to build the correct version of the binary.
5. Write upgrade instructions for validators (you can find many good examples of this for most chains already, and it should be relatively easy to reuse some of those - just make sure you test all the commands yourself :))
6. Do an upgrade on a testnet (if you don't have a testnet with your current version running, I would recommend tagging your current version and spinning one up with a few validators) and connect with other testnets you want to connect to once live. It also gives you the ability to gain knowledge on tooling and how it will look like in production.
7. Upgrade on mainnet when you feel good about the testnet

There are some high-level information available here: https://tutorials.cosmos.network/academy/2-cosmos-concepts/16-migrations.html

Some other points on upgrades:

Coordinating with validators for the upgrade.
It is good to have a good coordination channel with validators. Make sure you can reach them. The most common approach is to have a discord server with a dedicated validator role (and maybe testnet validator role) so you can tag validators when there are upgrades, governance proposals or issues. The key here is to have most validators in there. If you don't it might be hard to do the upgrade (and you don't want to end up in a state where the chain doesn't start up again because the validators have not upgraded their binary).

Proposing and implementing a software upgrade through governance.
There is some information available here https://medium.com/web3-surfers/cosmos-dev-series-cosmos-sdk-based-blockchain-upgrade-b5e99181554c and here https://docs.cosmos.network/v0.50/build/modules/upgrade

The cosmos hub docs also have some useful details on this: https://hub.cosmos.network/main/governance/submitting.html

If you write an upgrade test suite, you should get a good idea on the process.

3 Are there any best practices or potential pitfalls to be aware of during this process?

I think reading the documentation for both Cosmos SDK and IBC is the best starting point, and if you test it properly, it should be fairly smooth.

I already mentioned the communication with validators, so having that set up well up-front is going to reduce the potential for issues. In addition, I would recommend making sure you have some experienced validators in your set, as they have done this many times and can potentially help out if there are any issues.

4 What testing strategies do you recommend post-upgrade?

Ideally, you should have a good idea of this from the testnet phase already. Look into relaying with both the go relayer (http://github.com/cosmos/relayer) and hermes (http://hermes.informal.systems).

I hope this helps :)

[Srutip04]

@gjermundgaraba I'm looking to upgrade cosmos sdk to a newer version. Since I'm quite new to Cosmos, I was wondering if you could provide some guidance or resources on how to go about upgrading the SDK. Any tutorials, documentation, or tips would be really helpful.

[gjermundgaraba]

The Cosmos SDK has documentation for the code level changes that needs to be done: https://github.com/cosmos/cosmos-sdk/blob/main/UPGRADING.md

If you change to the tag or branch of the SDK release version you are planning to upgrade to, you will find upgrade instructions in the file linked :)