feat: Implement rollback command #11179

yihuang · 2022-02-14T11:13:35Z

Description

~~Currently it has panic: StoreBlockHeight (6) > StateBlockHeight + 1 (5) error when start after rollback.~~ fixed after update tendermint.

Author Checklist

All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.

I have...

Reviewers Checklist

All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.

I have...

confirmed the correct type prefix in the PR title
confirmed ! in the type prefix if API or client breaking change
confirmed all author checklist items have been addressed
reviewed state machine logic
reviewed API design and naming
reviewed documentation is accurate
reviewed tests and test coverage
manually tested (if applicable)

store/rootmulti/store.go

robert-zaremba · 2022-02-16T13:43:52Z

store/rootmulti/store.go

+	}
+	rs.pruneStores()
+
+	if i < target {


Shouldn't this be <=?

Shouldn't this be <=?

no, it's to check if latest height is changed, I just fixed it.

robert-zaremba · 2022-02-16T13:44:30Z

server/rollback.go

+			height, hash, err := tmcmd.RollbackState(ctx.Config)
+			if err != nil {
+				return fmt.Errorf("failed to rollback tendermint state: %w", err)
+			}


hmm, we need to be sure that both tendermint an SDK will rollback to the same version. In RollbackToVersion we can essentially rollback to an older version than the target.

RollbackToVersion only prune heights that are bigger than the height returned by tendermint's rollback.

bradyjoestar · 2022-02-17T04:01:02Z

I run a 4-peers simd cluster, rollbacked the peer3 data and restarted it.
The rollback cmd is:

simd  rollback  --home ~/cosmos_testnet/validators/validator3/simd/

It reported the following errors:


3:57AM INF starting ABCI with Tendermint
3:57AM INF starting service impl=multiAppConn module=proxy service=multiAppConn
3:57AM INF starting service connection=query impl=localClient module=abci-client service=localClient
3:57AM INF starting service connection=snapshot impl=localClient module=abci-client service=localClient
3:57AM INF starting service connection=mempool impl=localClient module=abci-client service=localClient
3:57AM INF starting service connection=consensus impl=localClient module=abci-client service=localClient
3:57AM INF starting service impl=EventBus module=events service=EventBus
3:57AM INF starting service impl=PubSub module=pubsub service=PubSub
3:57AM INF starting service impl=IndexerService module=txindex service=IndexerService
3:57AM INF ABCI Handshake App Info hash="����\v���Ѫz�\bC�%�0\b��I�Y����b���" height=5 module=consensus protocol-version=0 software-version=0.43.0-beta1-777-g1408c6d21
3:57AM INF ABCI Replay Blocks appHeight=5 module=consensus stateHeight=4 storeHeight=5
3:57AM INF Replay last block using mock app module=consensus
Error: error during handshake: error on replay: wrong Block.Header.AppHash.  Expected 2DB43C7B3D3556C02EDACCBEB650717FD60819CF066AF07E06407638BA9651F3, got 1504D73ED2FC4FBFB759C2948670A2DC8A34A20201256BB3FB2BEE5FD1199E2E
Usage:
  simd start [flags]

Flags:
      --abci string                                     specify abci transport (socket | grpc) (default "socket")
      --address string                                  Listen address (default "tcp://0.0.0.0:26658")
      --blocksync.enable                                enable fast blockchain syncing (default true)
      --consensus.create-empty-blocks                   set this to false to only produce blocks when there are txs or when the AppHash changes (default true)
      --consensus.create-empty-blocks-interval string   the possible interval between empty blocks (default "0s")

yihuang · 2022-02-17T07:46:06Z

I run a 4-peers simd cluster, rollbacked the peer3 data and restarted it.

I just fixed a trivial issue, can you try again.

PS: can you share your script to setup devnet, my script don't work with tendermint 0.35 somehow, and I haven't figured it out.

I always get this error when I backport this patch to 0.45.x and test with that:

"StoreBlockHeight (9) > StateBlockHeight + 1 (8)"

bradyjoestar · 2022-02-17T08:43:12Z

PS: can you share your script to setup devnet, my script don't work with tendermint 0.35 somehow, and I haven't figured it out.

yeah, I will try it later.
Here is the setup scripts.
https://github.com/bradyjoestar/cosmos_rollback

yihuang · 2022-02-17T10:15:11Z

PS: can you share your script to setup devnet, my script don't work with tendermint 0.35 somehow, and I haven't figured it out.

yeah, I will try it later. Here is the setup scripts. https://github.com/bradyjoestar/cosmos_rollback

after fixing the rollback in tendermint: tendermint/tendermint#7837, it works now.

go.mod

tac0turtle · 2022-03-02T18:08:18Z

@yihuang 0.35.2 was released with the patch, can you update then we merge this

JayT106 · 2022-03-02T21:46:50Z

store/rootmulti/store.go

@@ -905,6 +905,30 @@ func (rs *Store) buildCommitInfo(version int64) *types.CommitInfo {
 	}
 }

+// RollbackToVersion delete the versions after `target` and update the latest version.
+func (rs *Store) RollbackToVersion(target int64) int64 {
+	if target < 0 {


the input target should be always greater than 0 right? the State.InitialHeight will be 1.

yes, I think so, just for safety here.

Closes: cosmos#10281 fix tendermint rollback changelog update tendermint to recent v0.35.x branch

yihuang · 2022-03-03T01:24:29Z

@yihuang 0.35.2 was released with the patch, can you update then we merge this

done

yihuang · 2022-03-03T01:25:16Z

@yihuang 0.35.2 was released with the patch, can you update then we merge this

shall we backport this to 0.45?

Closes: #10281 fix tendermint rollback changelog update tendermint to recent v0.35.x branch (cherry picked from commit 8296ad9) # Conflicts: # CHANGELOG.md # go.mod # go.sum

alexanderbez · 2022-03-03T16:40:22Z

So there aren't any tests for this. Was this tested to actually work?

Because in my attempt I came across IAVL issues: #11233 (comment). Did you not come across this?

This should not have been merged w/o unit tests for the multi-store IMO.

yihuang · 2022-03-04T00:19:56Z

So there aren't any tests for this. Was this tested to actually work?

Because in my attempt I came across IAVL issues: #11233 (comment). Did you not come across this?

This should not have been merged w/o unit tests for the multi-store IMO.

I did the basic test manually:

start a devnet
stop devnet at a certain height
call ./simd rollback --home path/to/node0, it show the height - 1.
start devnet and observe node0's log, there's:

executed block height=5
stopping...
starting...
ABCI Replay Blocks appHeight=4 module=consensus stateHeight=4 storeHeight=5
executed block height=5
...
executed block height=6
...

alexanderbez · 2022-03-04T03:29:54Z

I don't see how that's possible. IAVL doesn't let you prune/delete the current version (height).

tac0turtle · 2022-03-07T10:45:51Z

@alexanderbez I was able to test this successfully locally on a chain that I submitted txs to in order to change the app hash a few times.

Closes: #10281 fix tendermint rollback changelog update tendermint to recent v0.35.x branch (cherry picked from commit 8296ad9) # Conflicts: # CHANGELOG.md # go.mod # go.sum Co-authored-by: yihuang <huang@crypto.com>

alexanderbez · 2022-03-08T15:07:54Z

That doesn't this should've been merged w/o ms unit tests.

yihuang · 2022-03-08T16:32:12Z

That doesn't this should've been merged w/o ms unit tests.

#11333

Closes: cosmos#10281 fix tendermint rollback changelog update tendermint to recent v0.35.x branch (cherry picked from commit 8296ad9) Co-authored-by: yihuang <huang@crypto.com>

Closes: cosmos#10281 fix tendermint rollback changelog update tendermint to recent v0.35.x branch (cherry picked from commit 8296ad9) # Conflicts: # CHANGELOG.md # go.mod # go.sum Co-authored-by: yihuang <huang@crypto.com>

Closes: cosmos#10281 fix tendermint rollback changelog update tendermint to recent v0.35.x branch (cherry picked from commit 8296ad9) Co-authored-by: yihuang <huang@crypto.com>

Closes: cosmos#10281 fix tendermint rollback changelog update tendermint to recent v0.35.x branch (cherry picked from commit 8296ad9) # Conflicts: # CHANGELOG.md # go.mod # go.sum Co-authored-by: yihuang <huang@crypto.com>

yihuang requested review from aaronc and alexanderbez as code owners February 14, 2022 11:13

github-actions bot added the C:Store label Feb 14, 2022

yihuang marked this pull request as draft February 14, 2022 11:21

yihuang force-pushed the rollback-cmd branch from 0b780af to 1408c6d Compare February 14, 2022 11:22

robert-zaremba reviewed Feb 16, 2022

View reviewed changes

store/rootmulti/store.go Outdated Show resolved Hide resolved

robert-zaremba reviewed Feb 16, 2022

View reviewed changes

yihuang force-pushed the rollback-cmd branch from c90ea14 to 7975cd3 Compare February 17, 2022 06:44

yihuang force-pushed the rollback-cmd branch from 7975cd3 to 18d9acd Compare February 17, 2022 10:12

yihuang changed the title ~~Implement rollback command~~ feat: Implement rollback command Feb 17, 2022

yihuang commented Feb 17, 2022

View reviewed changes

go.mod Outdated Show resolved Hide resolved

yihuang mentioned this pull request Feb 17, 2022

fix app hash in state rollback tendermint/tendermint#7837

Merged

yihuang force-pushed the rollback-cmd branch 2 times, most recently from a1a98c5 to 480b7d4 Compare February 21, 2022 02:56

yihuang marked this pull request as ready for review February 21, 2022 03:00

yihuang mentioned this pull request Feb 21, 2022

feat: rollback state CLI command #11233

Closed

19 tasks

yihuang force-pushed the rollback-cmd branch from 480b7d4 to c74ced1 Compare February 26, 2022 03:36

JayT106 reviewed Mar 2, 2022

View reviewed changes

Implement rollback command

6e8f78d

Closes: cosmos#10281 fix tendermint rollback changelog update tendermint to recent v0.35.x branch

yihuang force-pushed the rollback-cmd branch from c74ced1 to 6e8f78d Compare March 3, 2022 01:21

tac0turtle added the backport/0.45.x label Mar 3, 2022

tac0turtle approved these changes Mar 3, 2022

View reviewed changes

tac0turtle merged commit 8296ad9 into cosmos:master Mar 3, 2022

mergify bot pushed a commit that referenced this pull request Mar 3, 2022

Implement rollback command (#11179)

09648a0

Closes: #10281 fix tendermint rollback changelog update tendermint to recent v0.35.x branch (cherry picked from commit 8296ad9) # Conflicts: # CHANGELOG.md # go.mod # go.sum

mergify bot mentioned this pull request Mar 3, 2022

feat: Implement rollback command (backport #11179) #11314

Merged

yihuang deleted the rollback-cmd branch March 3, 2022 15:00

yihuang added a commit that referenced this pull request Mar 3, 2022

Implement rollback command (#11179)

0664b04

Closes: #10281 fix tendermint rollback changelog update tendermint to recent v0.35.x branch (cherry picked from commit 8296ad9) # Conflicts: # CHANGELOG.md # go.mod # go.sum

yihuang mentioned this pull request Mar 11, 2022

The rollback command is not well tested yet #11333

Closed

4 tasks

iramiller mentioned this pull request Apr 11, 2022

Add support for rollback command provenance-io/provenance#780

Closed

4 tasks

yihuang mentioned this pull request Nov 17, 2023

Add public api BlockStore.Rollback cometbft/cometbft#1610

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Implement rollback command #11179

feat: Implement rollback command #11179

yihuang commented Feb 14, 2022 •

edited

Loading

robert-zaremba Feb 16, 2022

yihuang Feb 17, 2022 •

edited

Loading

robert-zaremba Feb 16, 2022

yihuang Feb 17, 2022

bradyjoestar commented Feb 17, 2022

yihuang commented Feb 17, 2022 •

edited

Loading

bradyjoestar commented Feb 17, 2022

yihuang commented Feb 17, 2022

tac0turtle commented Mar 2, 2022

JayT106 Mar 2, 2022 •

edited

Loading

yihuang Mar 3, 2022

yihuang commented Mar 3, 2022

yihuang commented Mar 3, 2022

alexanderbez commented Mar 3, 2022

yihuang commented Mar 4, 2022 •

edited

Loading

alexanderbez commented Mar 4, 2022

tac0turtle commented Mar 7, 2022

alexanderbez commented Mar 8, 2022

yihuang commented Mar 8, 2022

feat: Implement rollback command #11179

feat: Implement rollback command #11179

Conversation

yihuang commented Feb 14, 2022 • edited Loading

Description

Author Checklist

Reviewers Checklist

robert-zaremba Feb 16, 2022

Choose a reason for hiding this comment

yihuang Feb 17, 2022 • edited Loading

Choose a reason for hiding this comment

robert-zaremba Feb 16, 2022

Choose a reason for hiding this comment

yihuang Feb 17, 2022

Choose a reason for hiding this comment

bradyjoestar commented Feb 17, 2022

yihuang commented Feb 17, 2022 • edited Loading

bradyjoestar commented Feb 17, 2022

yihuang commented Feb 17, 2022

tac0turtle commented Mar 2, 2022

JayT106 Mar 2, 2022 • edited Loading

Choose a reason for hiding this comment

yihuang Mar 3, 2022

Choose a reason for hiding this comment

yihuang commented Mar 3, 2022

yihuang commented Mar 3, 2022

alexanderbez commented Mar 3, 2022

yihuang commented Mar 4, 2022 • edited Loading

alexanderbez commented Mar 4, 2022

tac0turtle commented Mar 7, 2022

alexanderbez commented Mar 8, 2022

yihuang commented Mar 8, 2022

yihuang commented Feb 14, 2022 •

edited

Loading

yihuang Feb 17, 2022 •

edited

Loading

yihuang commented Feb 17, 2022 •

edited

Loading

JayT106 Mar 2, 2022 •

edited

Loading

yihuang commented Mar 4, 2022 •

edited

Loading