Upgrade to tm version 23.0. #1927

ValarDragon · 2018-08-06T23:02:59Z

This is Ready for review! This upgrade meant we had to maintain a map from addr to pubkey. This also removes crypto.Signature, and changes abci time from int64 to time.Time.

Updated all relevant documentation (docs/)
Updated all relevant code comments - (I think so?)
Wrote tests - n/a?
Added entries in PENDING.md that include links to the relevant issue or PR that most accurately describes the change.

For Admin Use:

Added appropriate labels to PR (ex. wip, ready-for-review, docs)
Reviewers Assigned
Squashed all commits, uses message "Merge pull request #XYZ: [title]" (coding standards)

ValarDragon · 2018-08-06T23:57:38Z

There are the hanging issues, which I have no idea whats causing them. Then theres also these two issues:

--- FAIL: TestUnbondingPeriod (0.23s)
	Error Trace:	handler_test.go:528
	Error:      	Should be true
	Test:       	TestUnbondingPeriod
	Messages:   	expected an error
--- FAIL: TestRedelegationPeriod (0.24s)
	Error Trace:	handler_test.go:587
	Error:      	Should be true
	Test:       	TestRedelegationPeriod
	Messages:   	expected an error

Those two are probably caused by the time change.

ValarDragon · 2018-08-07T20:34:48Z

Hanging has now been fixed, its just halting now.

Found the cause of the halting. In slashing/tick.go we have a handleValidatorSignature which takes in the public key. However we changed the Signing Validator struct to just return the address. The application is meant to maintain an address to pubkey mapping. So I guess we need to do this in slashing. (I am confused about what the merits of maintaining this map are though / but thats a separate discussion: tendermint/tendermint#2177 (comment))

/cc @cwgoes @rigelrozanski

ValarDragon · 2018-08-08T05:39:14Z

Figuring out why the LCD tests were failing is quite difficult from our logs. We really need to reduce the log output there, I have no idea what the error is after looking at all of those messages. (I don't even know which of the tests is failing) I had to find this out via manually testing each test individually. (It was a rather simple fix, just finding the error was the hardest part)

Currently this doesn't delete from the address to pubkey map. I'm not sure what the best way to delete from this map is, since it has to be deleted when a "CompleteUnbonding" transaction goes through. (As we can still receive slashing evidence until that point) For purposes of getting the next gaia release, I think we can punt on figuring this part out. (We may deem that its unnecessary after tendermint/tendermint#2177)

codecov · 2018-08-08T06:00:46Z

Codecov Report

Merging #1927 into develop will decrease coverage by <.01%.
The diff coverage is 65.21%.

@@             Coverage Diff             @@
##           develop    #1927      +/-   ##
===========================================
- Coverage    64.86%   64.86%   -0.01%     
===========================================
  Files          114      115       +1     
  Lines         6837     6862      +25     
===========================================
+ Hits          4435     4451      +16     
- Misses        2120     2127       +7     
- Partials       282      284       +2

cwgoes · 2018-08-08T10:35:18Z

The application is meant to maintain an address to pubkey mapping. So I guess we need to do this in slashing.

I don't think we should maintain such a map unless absolutely necessary, I think we should only use the validator address and treat it as an opaque key.

cwgoes · 2018-08-08T23:10:21Z

baseapp/baseapp.go

@@ -412,11 +413,7 @@ func (app *BaseApp) CheckTx(txBytes []byte) (res abci.ResponseCheckTx) {
 		Log:       result.Log,
 		GasWanted: result.GasWanted,
 		GasUsed:   result.GasUsed,
-		Fee: cmn.KI64Pair{


Why are we getting rid of this? Presumably Tendermint needs to know about the fee?

Fees were removed.
from the changelog:
[abci] Removed Fee from ResponseDeliverTx and ResponseCheckTx

Oh, should we add priority then?

@ValarDragon thoughts on the above?

No reason to add priority right now. We're not ready to support that at the mempool level, and we won't be for awhile. We discussed that priority is something we want to add as a way to test protobuf upgradability on a test net, not right now.

OK, I think it's not necessary to delay ABCI updates because Tendermint won't use the value yet, but nbd.

cwgoes · 2018-08-08T23:15:28Z

crypto/encode_test.go

 	// Make sure we have the expected length.
 	if size != -1 {
-		require.Equal(t, size, len(bz), "Amino binary size mismatch")
+		assert.Equal(t, size, len(bz), "Amino binary size mismatch")


Why assert instead of require? Wouldn't we want this to fail immediately?

I was changing this to match whats in tendermint/crypto right now. I don't really have a preference here.

I think it's most important to be consistent -- it seems in SDK-land, the defacto call is require.

Its to use require unless there is any reason not to. There is good reason to not use it here if we had more pubkey types, but I guess we can change it when that happens.

alexanderbez

Looks great @ValarDragon. Left a few minor remarks 👍

alexanderbez · 2018-08-09T18:27:58Z

PENDING.md

+^--- PENDING wasn't purged on sdk v0.23.0 release.
+
+BREAKING CHANGES
+* Update to tendermint v0.23.0. This involves removing crypto.Pubkey, 


Can we add the PR reference here (preferably with a link por favor)?

I'm not sure why the PR is helpful. The tendermint changelog is what is helpful, I can link that?

But the SDK changed and that's important to note where/how it happened.

I guess this is a downside of our pending system, the commit with all the changes can't be obtained from the file. I'll link the PR then.

Yeah, I'd just would like to see a CHANGELOG and be able to go to the issue and/or PR that made the relevant changes.

alexanderbez · 2018-08-09T18:29:53Z

x/slashing/genesis.go

+	"github.com/cosmos/cosmos-sdk/x/stake/types"
+)
+
+// InitGenesis initializes the keeper's address to pubkey map


Let's add punctuation here.

If we want to enforce this stuff, we should have it as output from a linter. I'll change this, but #postlaunch we should make the linter say this.

I agree 👍 .

alexanderbez · 2018-08-09T18:33:04Z

x/slashing/keeper.go

+	}
+}
+
+// TODO: Make a method to remove the pubkey from the map when a validator is unbonded.


I think this is pretty trivial, no? Why not just add the call to Keeper#unbondValidator?

The actual method on the slashing keeper is trivial. Making this work when a validator is unbonded is non-trivial. We need to have a way for the staking keeper to talk to the slashing keeper. The method here isn't added, since we should hold off until we figure that out.

Given the discussion in tendermint/tendermint#2177, we may just want to not do this at all, so no point figuring it out. (We can live with one hashmap entry per every validator on the next testnet)

I see I see.

alexanderbez · 2018-08-09T18:35:50Z

x/slashing/keeper.go

+}
+
+func (k Keeper) getPubkey(address crypto.Address) (crypto.PubKey, error) {
+	addr := new([tmhash.Size]byte)


Is there a reason we allocate a pointer instead of just:

var addr [tmhash.Size]byte copy(addr[:], address) // ...

var addr [tmhash.Size]byte ~~is equivalent to what new does. It just has more indirection. I don't know what you mean by "allocating a pointer", since that is used for var as well.~~

EDIT: Nevermind, your right. Var seems like the right way to go :), (forgot that var didn't make a pointer)

new returns a memory address -- that way you don't have to do *addr but just addr. No big deal though.

ValarDragon · 2018-08-09T19:28:02Z

#1927 (comment)

Lets continue discussion of that in tendermint/tendermint#2177. Bucky was suggesting that its important to maintain referrence to the pubkey, and I think we should only use the pubkey.

ValarDragon · 2018-08-09T19:51:10Z

Currently the addr-> pubkey map isn't persisted in state. (Thanks for pointing that out chris), that needs to be finished before this can be merged.

…sion

alexanderbez

LGTM @ValarDragon 👍

…le maps, we have to change from what this commit is doing.

ValarDragon · 2018-08-09T21:54:39Z

This should be good to merge now :) (pending final review)

cwgoes · 2018-08-10T08:00:01Z

x/slashing/keeper.go

+	for i := 0; i < len(vals); i++ {
+		val := vals[i]
+		pubkey, err := tmtypes.PB2TM.PubKey(val.PubKey)
+		if err != nil {


Should we panic here?

cwgoes · 2018-08-10T08:01:22Z

x/slashing/keeper.go

+}
+
+// TODO: Make a method to remove the pubkey from the map when a validator is unbonded.
+func (k Keeper) addPubkey(ctx sdk.Context, pubkey crypto.PubKey, del bool) {


Having a function called addPubkey which takes a delete boolean is a bit confusing, why not just have addPubkey and deletePubkey?

At minimum change the function name, maybe to addOrRemovePubkey

cwgoes · 2018-08-10T08:01:31Z

x/slashing/keeper.go

+func (k Keeper) getPubkey(ctx sdk.Context, address crypto.Address) (crypto.PubKey, error) {
+	store := ctx.KVStore(k.storeKey)
+	var pubkey crypto.PubKey
+	err := k.cdc.UnmarshalBinary(store.Get(getAddrPubkeyRelationKey(address)), &pubkey)


MustUnmarshalBinary

(unless you wanted the custom error, in which case disregard)

I do think a custom error message is preferrable

cwgoes

LGTM mostly, a few minor nits.

cwgoes

utACK, we should definitely test any Tendermint-23 relevant changes on an internal testnet.

ValarDragon · 2018-08-10T13:18:26Z

That basically means testing liveness slashing, and unbonding /rebonding due to time changes. (I'm not worried about the crypto.Signature change breaking stuff)

Also yeah, was in a hurry when writing that last commit, forgot to test

ebuchman

Thanks Dev.

Note gaiadebug binary got accidentally checked in

We might also want our own Time function that returns the .UTC for us automatically and so we don't have to pass in nanoseconds.

Also I wonder when is it better to use a new type for []byte, eg. type Signature []byte ?

ebuchman · 2018-08-12T06:57:49Z

x/slashing/keeper.go

+}
+
+func getAddrPubkeyRelationKey(address []byte) []byte {
+	return append([]byte{0x03}, address...)


magic number

ebuchman · 2018-08-12T06:59:42Z

x/slashing/signing_info_test.go

@@ -15,15 +16,15 @@ func TestGetSetValidatorSigningInfo(t *testing.T) {
 	newInfo := ValidatorSigningInfo{
 		StartHeight:         int64(4),
 		IndexOffset:         int64(3),
-		JailedUntil:         int64(2),
+		JailedUntil:         time.Unix(2, 0),


The utc was in some places due to time defaulting to the 0 value. (and not everywhere else)

It is kind of sub ideal though, but i thought just using utc was preferable to going through a separate initializer for those tests when initializing validators.

ebuchman · 2018-08-12T07:02:43Z

x/stake/types/params.go


 	sdk "github.com/cosmos/cosmos-sdk/types"
 )

 // defaultUnbondingTime reflects three weeks in seconds as the default
 // unbonding time.
-const defaultUnbondingTime int64 = 60 * 60 * 24 * 3
+const defaultUnbondingTime time.Duration = 60 * 60 * 24 * 3 * time.Second


note it's currently 3 days not 3 weeks

ebuchman · 2018-08-12T07:08:41Z

We also need to update the spec about the addr->pubkey map since it's being persisted.

Though looks like we might revert the need for this depending on what happens with tendermint/tendermint#2177

Start upgrade. Currently go test ./... hangs.

7410bf6

ValarDragon requested review from cwgoes, ebuchman, rigelrozanski and zramsay as code owners August 6, 2018 23:02

(squash this) Fix staking tests

c62380d

wip

73b4bdd

ebuchman mentioned this pull request Aug 8, 2018

v0.24 release #1935

Closed

6 tasks

ValarDragon added 2 commits August 7, 2018 19:37

note what changes need to be made to make this work on tm v0.23.0

57693b9

Fix addr -> pubkey map

41d0355

ValarDragon changed the title ~~WIP: Upgrade to tm version 23.0.~~ Upgrade to tm version 23.0. Aug 8, 2018

ValarDragon added the ready-for-review label Aug 8, 2018

ValarDragon force-pushed the dev/upgrade_tm_version branch from 0f11891 to 56d8eac Compare August 8, 2018 04:55

cleanup code

2f85fff

ValarDragon force-pushed the dev/upgrade_tm_version branch from 56d8eac to 2f85fff Compare August 8, 2018 04:56

ValarDragon added 2 commits August 8, 2018 00:14

Fix slashing test failures except for begin blocker

4ee35d4

fix all slashing tests

d130bbf

fix lcd tests

a107a80

cwgoes mentioned this pull request Aug 8, 2018

Comments on ADR 18 (ABCI validators) tendermint/tendermint#2177

Closed

cwgoes self-assigned this Aug 8, 2018

cwgoes reviewed Aug 8, 2018

View reviewed changes

alexanderbez reviewed Aug 9, 2018

View reviewed changes

Address PR comments

7b52940

ValarDragon added 2 commits August 9, 2018 15:00

add link to changelog.

794a699

Merge remote-tracking branch 'origin/develop' into dev/upgrade_tm_ver…

d2acf55

…sion

alexanderbez approved these changes Aug 9, 2018

View reviewed changes

ValarDragon added 2 commits August 9, 2018 15:41

(wip) start making addrToPubkey map persisted. Since amino can't hand…

a27a43b

…le maps, we have to change from what this commit is doing.

Use the correct method of storing a map

f44a487

cwgoes reviewed Aug 10, 2018

View reviewed changes

ValarDragon and others added 2 commits August 10, 2018 06:40

(squash this) address PR comments

c4b6c58

Did you run 'make'?

0e85025

cwgoes approved these changes Aug 10, 2018

View reviewed changes

Merge branch 'develop' into dev/upgrade_tm_version

161dfaa

ebuchman approved these changes Aug 12, 2018

View reviewed changes

ebuchman mentioned this pull request Aug 12, 2018

x/slashing: use const for key prefixes #1990

Closed

remove gaiadebug binary

7322cda

ebuchman merged commit b2a4aec into develop Aug 12, 2018

ebuchman deleted the dev/upgrade_tm_version branch August 12, 2018 07:33

Upgrade to tm version 23.0. #1927

Upgrade to tm version 23.0. #1927

Conversation

ValarDragon commented Aug 6, 2018 • edited Loading

ValarDragon commented Aug 6, 2018

ValarDragon commented Aug 7, 2018 • edited Loading

ValarDragon commented Aug 8, 2018 • edited Loading

codecov bot commented Aug 8, 2018 • edited Loading

Codecov Report

cwgoes commented Aug 8, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cwgoes Aug 10, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexanderbez left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ValarDragon Aug 9, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ValarDragon Aug 9, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ValarDragon commented Aug 9, 2018

ValarDragon commented Aug 9, 2018

alexanderbez left a comment

Choose a reason for hiding this comment

ValarDragon commented Aug 9, 2018 • edited Loading

Choose a reason for hiding this comment

cwgoes Aug 10, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cwgoes left a comment

Choose a reason for hiding this comment

cwgoes left a comment

Choose a reason for hiding this comment

ValarDragon commented Aug 10, 2018 • edited Loading

ebuchman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ebuchman commented Aug 12, 2018

ValarDragon commented Aug 6, 2018 •

edited

Loading

ValarDragon commented Aug 7, 2018 •

edited

Loading

ValarDragon commented Aug 8, 2018 •

edited

Loading

codecov bot commented Aug 8, 2018 •

edited

Loading

cwgoes commented Aug 8, 2018 •

edited

Loading

cwgoes Aug 10, 2018 •

edited

Loading

ValarDragon Aug 9, 2018 •

edited

Loading

ValarDragon Aug 9, 2018 •

edited

Loading

ValarDragon commented Aug 9, 2018 •

edited

Loading

cwgoes Aug 10, 2018 •

edited

Loading

ValarDragon commented Aug 10, 2018 •

edited

Loading