Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Robert/addr memory leak #8717

Merged
merged 15 commits into from
Mar 3, 2021
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 16 additions & 34 deletions types/address.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ import (
"github.com/cosmos/cosmos-sdk/types/address"
"github.com/cosmos/cosmos-sdk/types/bech32"
sdkerrors "github.com/cosmos/cosmos-sdk/types/errors"
"github.com/hashicorp/golang-lru/simplelru"
)

const (
Expand Down Expand Up @@ -158,12 +159,7 @@ func (aa AccAddress) Equals(aa2 Address) bool {

// Returns boolean for whether an AccAddress is empty
func (aa AccAddress) Empty() bool {
if aa == nil {
return true
}

aa2 := AccAddress{}
return bytes.Equal(aa.Bytes(), aa2.Bytes())
return aa == nil || len(aa) == 0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

}

// Marshal returns the raw address bytes. It is needed for protobuf
Expand Down Expand Up @@ -239,37 +235,33 @@ func (aa AccAddress) Bytes() []byte {

// AccAddress.String() is expensive and if unoptimized dominantly showed up in profiles,
// yet has no mechanisms to trivially cache the result given that AccAddress is a []byte type.
// With the string interning below, we are able to achieve zero cost allocations for string->[]byte
// because the Go compiler recognizes the special case of map[string([]byte)] when used exactly
// in that pattern. See https://github.com/cosmos/cosmos-sdk/issues/8693.
var addMu sync.Mutex
var addrStrMemoize = make(map[string]string)
var addrStrMemoize, _ = simplelru.NewLRU(1000, nil)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This initialization should be in an init() function, so that we can check the error returned and panic() if it is != nil.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was following the current design. That being said I agree with you - I will use init.

tac0turtle marked this conversation as resolved.
Show resolved Hide resolved

// String implements the Stringer interface.
func (aa AccAddress) String() (str string) {
addMu.Lock()
defer addMu.Unlock()

if str, ok := addrStrMemoize[string(aa)]; ok {
return str
func (aa AccAddress) String() string {
var key = string(aa)
odeke-em marked this conversation as resolved.
Show resolved Hide resolved
if str, ok := addrStrMemoize.Get(key); ok {
return str.(string)
}

// Otherwise cache it for later memoization.
defer func() {
addrStrMemoize[string(aa)] = str
}()

if aa.Empty() {
addMu.Lock()
addrStrMemoize.Add(key, "")
addMu.Unlock()
return ""
}

bech32PrefixAccAddr := GetConfig().GetBech32AccountAddrPrefix()

bech32Addr, err := bech32.ConvertAndEncode(bech32PrefixAccAddr, aa.Bytes())
bech32Addr, err := bech32.ConvertAndEncode(bech32PrefixAccAddr, aa)
alessio marked this conversation as resolved.
Show resolved Hide resolved
if err != nil {
panic(err)
}

addMu.Lock()
addrStrMemoize.Add(key, bech32Addr)
addMu.Unlock()
return bech32Addr
}

Expand Down Expand Up @@ -332,12 +324,7 @@ func (va ValAddress) Equals(va2 Address) bool {

// Returns boolean for whether an AccAddress is empty
func (va ValAddress) Empty() bool {
if va == nil {
return true
}

va2 := ValAddress{}
return bytes.Equal(va.Bytes(), va2.Bytes())
return va == nil || len(va) == 0
}

// Marshal returns the raw address bytes. It is needed for protobuf
Expand Down Expand Up @@ -492,12 +479,7 @@ func (ca ConsAddress) Equals(ca2 Address) bool {

// Returns boolean for whether an ConsAddress is empty
func (ca ConsAddress) Empty() bool {
if ca == nil {
return true
}

ca2 := ConsAddress{}
return bytes.Equal(ca.Bytes(), ca2.Bytes())
return ca == nil || len(ca) == 0
}

// Marshal returns the raw address bytes. It is needed for protobuf
Expand Down
12 changes: 12 additions & 0 deletions types/address_bench_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,18 @@ import (
"github.com/cosmos/cosmos-sdk/types"
)

func BenchmarkAccAddressString(b *testing.B) {
b.ReportAllocs()
pkBz := make([]byte, ed25519.PubKeySize)
pk := &ed25519.PubKey{Key: pkBz}
aa := pk.Address()
var str string
for i := 0; i < b.N; i++ {
str = aa.String()
}
require.NotEmpty(b, str)
alessio marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To better get a good benchmark in which code won't be optimized away, use a global variable that's an interface e.g.

var sink, revert interface{}
...
for i := 0; i < b.N; i++ {
   result := ...
   sink = result
}
if  sink == nil {
   b.Fatal("benchmark never ran")
}
sink = revert

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean a variable outside of any function? We are not using it in other benchmarks.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well just because it wasn't done correctly before doesn't mean we shouldn't do it correctly, I certainly have submitted many other benchmarks where we set to a global sink and check that.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked it locally - it doesn't change anything. Are you sure we need to trick the benchmark code?
My only objection is that we make the whole thing less readable and more complex without any observable effect.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am sure about code at times getting optimized away and that being the only way to block that. Humbly, for a bit of clarity, I work on the Go project, across almost all the packages and I am bringing this advice from the many years of working on benchmarking related code :-)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. But why to obfuscate more if there is no difference (I verified it)? The require.NoEmpty already assures that value must not be ignored.

}

func BenchmarkBech32ifyPubKey(b *testing.B) {
b.ReportAllocs()
pkBz := make([]byte, ed25519.PubKeySize)
Expand Down