Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Atree register inlining for v1.0 #3048

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
d364afc
Update to new Atree API
fxamacker Jun 29, 2023
79a115d
Update to new Atree API
fxamacker Jul 7, 2023
caa71b7
Track resources with atree.ValueID instead of SlabID
fxamacker Jul 7, 2023
efc4100
Replace magic num to access Atree slab ID components
fxamacker Jul 7, 2023
2c7fb10
Integrate with atree inlining and deduplication
fxamacker Oct 19, 2023
c9f615b
Fix lint warnings
fxamacker Oct 19, 2023
829cbc0
Use UnreachableError in panic
fxamacker Oct 19, 2023
e88e39b
Simplify TypeInfo.Identifier()
fxamacker Oct 19, 2023
1130343
Reformat with extra lines for readability
fxamacker Oct 19, 2023
8d4310f
meter allocations of array static types
turbolent Oct 19, 2023
b339351
disable atree validation for memory metering tests, adjust expected v…
turbolent Oct 19, 2023
02c96a8
pass proper location range
turbolent Oct 20, 2023
5fa310c
simplify dictionary iterator to just support key iteration, which all…
turbolent Oct 20, 2023
b875ba1
use type ID constructor function
turbolent Oct 20, 2023
273e2c6
SomeStorable.Encode should not ever be called. use proper encode func…
turbolent Oct 20, 2023
4c75345
address TODOs: Identifier functions can use StaticType.ID, Copy can r…
turbolent Oct 20, 2023
766a6f8
Update to use latest Atree inlining
fxamacker Oct 22, 2023
7977031
Refactor SomeStorable encoding
fxamacker Oct 24, 2023
3a36523
Update to use latest Atree inlining
fxamacker Oct 24, 2023
5e2d3e4
lower size of generated containers
turbolent Oct 24, 2023
c9efb80
add tests for "moving" struct and composite between arrays, and mutat…
turbolent Oct 24, 2023
98e8a09
Cleanup and add more comments
fxamacker Oct 25, 2023
c316e18
Add another test for SomeValue encoding
fxamacker Oct 25, 2023
54a9289
Add smoke test flag smokeTestSeed
fxamacker Oct 25, 2023
e6bc174
Add reproducer for false alarm (unreachable slab)
fxamacker Oct 31, 2023
fd2567c
add atRoot parameter to Value.DeepRemove. only validate atree value i…
turbolent Oct 31, 2023
6d470cc
always perform atree value validation, only perform atree storage val…
turbolent Oct 31, 2023
ba3ccd9
Add reproducer for false alarm (unreachable slab)
fxamacker Nov 9, 2023
2f24479
Add `atRoot` parameter to Value.Transfer()
fxamacker Nov 9, 2023
eb897b2
Use Atree non-readonly iterators in Transfer()
fxamacker Nov 21, 2023
b41c0ea
Rename atRoot param for Transfer() and DeepRemove()
fxamacker Nov 21, 2023
1d38c99
Use hasNoParentContainer for inner value DeepRemove()
fxamacker Nov 21, 2023
d2b71e6
Update StorageMapKey funcs needed by atree iterators
fxamacker Jan 18, 2024
e185aad
update to latest atree
turbolent Jan 24, 2024
dbdb55f
Update to use latest Atree iterator API
fxamacker Jan 24, 2024
e959433
apply StorageIndex -> SlabIndex change to moved test ledger
turbolent Jan 26, 2024
6a87ada
remove accidentally re-added account reference value
turbolent Jan 26, 2024
b9acebe
fix iteration
turbolent Jan 26, 2024
0f13c24
adjust to API changes
turbolent Jan 26, 2024
027e3c0
skip test for now
turbolent Jan 26, 2024
137c8b1
Apply suggestions from code review
turbolent Jan 30, 2024
f9c1d68
Add DictionaryValue.IterateReadOnly()
fxamacker Jan 30, 2024
e7beed7
Fix panic in TestRehash
fxamacker Jan 30, 2024
d4f10b0
Fix lint warnings
fxamacker Jan 30, 2024
d0b144b
Fix entitlement migration
fxamacker Jan 30, 2024
0f46cb4
Fix migration and LegacyIntersectionType.Equal()
fxamacker Jan 30, 2024
c8b0110
Fix migration and LegacyPrimitiveStaticType.Equal()
fxamacker Jan 30, 2024
bedd498
adjust memory metering
turbolent Jan 30, 2024
b89fc51
InclusiveRangeStaticType does not need to implement atree.TypeInfo (yet)
turbolent Jan 31, 2024
6713e43
adjust test to new reference-to-resource semantics of Cadence 1.0
turbolent Jan 31, 2024
3f70345
Use atree readonly iterators when feasible
fxamacker Mar 12, 2024
0f59d17
Bump atree version for extra deduplication
fxamacker Mar 15, 2024
2cf9fce
Merge pull request #3178 from onflow/fxamacker/update-atree-register-…
fxamacker Mar 28, 2024
b68900d
Use atree non-readonly iterators in some places
fxamacker Apr 4, 2024
fccfa81
Merge pull request #3170 from onflow/fxamacker/use-atree-readonly-ite…
fxamacker Apr 4, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ require (
github.com/kr/pretty v0.3.1
github.com/leanovate/gopter v0.2.9
github.com/logrusorgru/aurora/v4 v4.0.0
github.com/onflow/atree v0.6.1-0.20230711151834-86040b30171f
github.com/onflow/atree v0.6.1-0.20240314011440-92714cac03fb
github.com/rivo/uniseg v0.4.4
github.com/schollz/progressbar/v3 v3.13.1
github.com/stretchr/testify v1.8.4
Expand Down
6 changes: 3 additions & 3 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -62,8 +62,8 @@ github.com/mattn/go-tty v0.0.3/go.mod h1:ihxohKRERHTVzN+aSVRwACLCeqIoZAWpoICkkvr
github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db h1:62I3jR2EmQ4l5rM/4FEfDWcRD+abF5XlKShorW5LRoQ=
github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db/go.mod h1:l0dey0ia/Uv7NcFFVbCLtqEBQbrT4OCwCSKTEv6enCw=
github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e h1:fD57ERR4JtEqsWbfPhv4DMiApHyliiK5xCTNVSPiaAs=
github.com/onflow/atree v0.6.1-0.20230711151834-86040b30171f h1:Z8/PgTqOgOg02MTRpTBYO2k16FE6z4wEOtaC2WBR9Xo=
github.com/onflow/atree v0.6.1-0.20230711151834-86040b30171f/go.mod h1:xvP61FoOs95K7IYdIYRnNcYQGf4nbF/uuJ0tHf4DRuM=
github.com/onflow/atree v0.6.1-0.20240314011440-92714cac03fb h1:9w+8wseSv7TZ9ikGfKSZ6yJifgYLhm717Zo0SVKpGEQ=
github.com/onflow/atree v0.6.1-0.20240314011440-92714cac03fb/go.mod h1:7YNAyCd5JENq+NzH+fR1ABUZVzbSq9dkt0+5fZH3L2A=
github.com/onflow/crypto v0.25.0 h1:BeWbLsh3ZD13Ej+Uky6kg1PL1ZIVBDVX+2MVBNwqddg=
github.com/onflow/crypto v0.25.0/go.mod h1:C8FbaX0x8y+FxWjbkHy0Q4EASCDR9bSPWZqlpCLYyVI=
github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA=
Expand Down Expand Up @@ -167,5 +167,5 @@ gopkg.in/check.v1 v1.0.0-20200902074654-038fdea0a05b h1:QRR6H1YWRnHb4Y/HeNFCTJLF
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
lukechampine.com/blake3 v1.1.7 h1:GgRMhmdsuK8+ii6UZFDL8Nb+VyMwadAgcJyfYHxG6n0=
lukechampine.com/blake3 v1.2.1 h1:YuqqRuaqsGV71BV/nm9xlI0MKUv4QC54jQnBChWbGnI=
rsc.io/pdf v0.1.1/go.mod h1:n8OzWcQ6Sp37PL01nO98y4iUCRdTGarVfzxY20ICaU4=
17 changes: 11 additions & 6 deletions migrations/entitlements/migration.go
Original file line number Diff line number Diff line change
Expand Up @@ -275,16 +275,21 @@ func ConvertValueToEntitlements(

var values []interpreter.Value

v.Iterate(inter, func(key, value interpreter.Value) (resume bool) {
values = append(values, key)
values = append(values, value)
return true
})
v.Iterate(
inter,
interpreter.EmptyLocationRange,
func(key, value interpreter.Value) (resume bool) {
values = append(values, key)
values = append(values, value)
return true
},
)

return interpreter.NewDictionaryValue(
return interpreter.NewDictionaryValueWithAddress(
inter,
interpreter.EmptyLocationRange,
dictionaryStaticType,
v.GetOwner(),
values...,
), nil

Expand Down
1 change: 1 addition & 0 deletions migrations/entitlements/migration_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -3034,6 +3034,7 @@ func TestRehash(t *testing.T) {
false,
nil,
nil,
true, // dictValue is standalone
),
)

Expand Down
3 changes: 2 additions & 1 deletion migrations/legacy_character_value.go
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,8 @@ func (v *LegacyCharacterValue) Transfer(
_ atree.Address,
remove bool,
storable atree.Storable,
_ map[atree.StorageID]struct{},
_ map[atree.ValueID]struct{},
_ bool,
) interpreter.Value {
if remove {
interpreter.RemoveReferencedSlab(storable)
Expand Down
33 changes: 33 additions & 0 deletions migrations/legacy_intersection_type.go
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,39 @@

var _ interpreter.StaticType = &LegacyIntersectionType{}

func (t *LegacyIntersectionType) Equal(other interpreter.StaticType) bool {
var otherTypes []*interpreter.InterfaceStaticType

switch other := other.(type) {

case *LegacyIntersectionType:
otherTypes = other.Types

case *interpreter.IntersectionStaticType:
otherTypes = other.Types

default:
return false

Check warning on line 48 in migrations/legacy_intersection_type.go

View check run for this annotation

Codecov / codecov/patch

migrations/legacy_intersection_type.go#L47-L48

Added lines #L47 - L48 were not covered by tests
}

if len(t.Types) != len(otherTypes) {
return false
}

Check warning on line 53 in migrations/legacy_intersection_type.go

View check run for this annotation

Codecov / codecov/patch

migrations/legacy_intersection_type.go#L52-L53

Added lines #L52 - L53 were not covered by tests

outer:
for _, typ := range t.Types {
for _, otherType := range otherTypes {
if typ.Equal(otherType) {
continue outer
}
}

return false

Check warning on line 63 in migrations/legacy_intersection_type.go

View check run for this annotation

Codecov / codecov/patch

migrations/legacy_intersection_type.go#L63

Added line #L63 was not covered by tests
}

return true
}

func (t *LegacyIntersectionType) ID() common.TypeID {
interfaceTypeIDs := make([]string, 0, len(t.Types))
for _, interfaceType := range t.Types {
Expand Down
14 changes: 14 additions & 0 deletions migrations/legacy_primitivestatic_type.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,20 @@

var _ interpreter.StaticType = LegacyPrimitiveStaticType{}

func (t LegacyPrimitiveStaticType) Equal(other interpreter.StaticType) bool {
switch other := other.(type) {

case LegacyPrimitiveStaticType:
return t == other

case interpreter.PrimitiveStaticType:
return t.PrimitiveStaticType == other

default:
return false

Check warning on line 45 in migrations/legacy_primitivestatic_type.go

View check run for this annotation

Codecov / codecov/patch

migrations/legacy_primitivestatic_type.go#L44-L45

Added lines #L44 - L45 were not covered by tests
}
}

func (t LegacyPrimitiveStaticType) ID() common.TypeID {
switch t.PrimitiveStaticType {
case interpreter.PrimitiveStaticTypeAuthAccount: //nolint:staticcheck
Expand Down
20 changes: 20 additions & 0 deletions migrations/legacy_reference_type.go
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,26 @@

var _ interpreter.StaticType = &LegacyReferenceType{}

// Equal() compares both value and type of t and other.
// LegacyReferenceType.Equal() is needed because Equal() in general
// compares values and their types. Embedded ReferenceStaticType.Equal()
// returns false when other is *LegacyReferenceType type.
func (t *LegacyReferenceType) Equal(other interpreter.StaticType) bool {
switch other := other.(type) {

case *LegacyReferenceType:
return t.Authorization.Equal(other.Authorization) &&
t.ReferencedType.Equal(other.ReferencedType)

case *interpreter.ReferenceStaticType:
return t.Authorization.Equal(other.Authorization) &&
t.ReferencedType.Equal(other.ReferencedType)

default:
return false

Check warning on line 53 in migrations/legacy_reference_type.go

View check run for this annotation

Codecov / codecov/patch

migrations/legacy_reference_type.go#L52-L53

Added lines #L52 - L53 were not covered by tests
}
}

func (t *LegacyReferenceType) ID() common.TypeID {
borrowedType := t.ReferencedType
return common.TypeID(
Expand Down
3 changes: 2 additions & 1 deletion migrations/legacy_string_value.go
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,8 @@ func (v *LegacyStringValue) Transfer(
_ atree.Address,
remove bool,
storable atree.Storable,
_ map[atree.StorageID]struct{},
_ map[atree.ValueID]struct{},
_ bool,
) interpreter.Value {
if remove {
interpreter.RemoveReferencedSlab(storable)
Expand Down
26 changes: 15 additions & 11 deletions migrations/migration.go
Original file line number Diff line number Diff line change
Expand Up @@ -227,18 +227,22 @@ func (m *StorageMigration) MigrateNestedValue(
// Read the keys first, so the iteration wouldn't be affected
// by the modification of the nested values.
var existingKeysAndValues []keyValuePair
dictionary.Iterate(m.interpreter, func(key, value interpreter.Value) (resume bool) {
existingKeysAndValues = append(
existingKeysAndValues,
keyValuePair{
key: key,
value: value,
},
)
dictionary.IterateReadOnly(
m.interpreter,
emptyLocationRange,
func(key, value interpreter.Value) (resume bool) {
existingKeysAndValues = append(
existingKeysAndValues,
keyValuePair{
key: key,
value: value,
},
)

// continue iteration
return true
})
// continue iteration
return true
},
)

for _, existingKeyAndValue := range existingKeysAndValues {
existingKey := existingKeyAndValue.key
Expand Down
2 changes: 2 additions & 0 deletions migrations/migration_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -482,6 +482,7 @@ func TestMultipleMigrations(t *testing.T) {
false,
nil,
nil,
true, // storedValue is standalone
)

inter.WriteStored(
Expand Down Expand Up @@ -621,6 +622,7 @@ func TestMigrationError(t *testing.T) {
false,
nil,
nil,
true, // storedValue is standalone
)

inter.WriteStored(
Expand Down
18 changes: 13 additions & 5 deletions migrations/statictypes/account_type_migration_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -687,6 +687,8 @@ func TestAccountTypeInNestedTypeValueMigration(t *testing.T) {
false,
nil,
nil,
true, // storedValue is standalone

)

inter.WriteStored(
Expand Down Expand Up @@ -833,6 +835,7 @@ func TestMigratingValuesWithAccountStaticType(t *testing.T) {
false,
nil,
nil,
true, // storedValue is standalone
)

inter.WriteStored(
Expand Down Expand Up @@ -976,6 +979,7 @@ func TestAccountTypeRehash(t *testing.T) {
false,
nil,
nil,
true, // dictValue is standalone
),
)

Expand Down Expand Up @@ -1035,11 +1039,15 @@ func TestAccountTypeRehash(t *testing.T) {
dictValue := storedValue.(*interpreter.DictionaryValue)

var existingKeys []interpreter.Value
dictValue.Iterate(inter, func(key, value interpreter.Value) (resume bool) {
existingKeys = append(existingKeys, key)
// continue iteration
return true
})
dictValue.Iterate(
inter,
interpreter.EmptyLocationRange,
func(key, value interpreter.Value) (resume bool) {
existingKeys = append(existingKeys, key)
// continue iteration
return true
},
)

for _, key := range existingKeys {
actual := dictValue.Remove(
Expand Down
3 changes: 3 additions & 0 deletions migrations/statictypes/intersection_type_migration_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -546,6 +546,7 @@ func TestIntersectionTypeRehash(t *testing.T) {
false,
nil,
nil,
true, // dictValue is standalone
),
)

Expand Down Expand Up @@ -710,6 +711,7 @@ func TestRehashNestedIntersectionType(t *testing.T) {
false,
nil,
nil,
true, // dictValue is standalone
),
)

Expand Down Expand Up @@ -849,6 +851,7 @@ func TestRehashNestedIntersectionType(t *testing.T) {
false,
nil,
nil,
true, // dictValue is standalone
),
)

Expand Down
4 changes: 4 additions & 0 deletions migrations/string_normalization/migration_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -245,6 +245,7 @@ func TestStringNormalizingMigration(t *testing.T) {
false,
nil,
nil,
true, // storedValue is standalone
)

inter.WriteStored(
Expand Down Expand Up @@ -382,6 +383,8 @@ func TestStringValueRehash(t *testing.T) {
false,
nil,
nil,
true, // dictValue is standalone

),
)

Expand Down Expand Up @@ -521,6 +524,7 @@ func TestCharacterValueRehash(t *testing.T) {
false,
nil,
nil,
true, // dictValue is standalone
),
)

Expand Down
6 changes: 5 additions & 1 deletion runtime/account_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -2277,9 +2277,13 @@ func TestRuntimePublicKeyPublicKeyField(t *testing.T) {
false,
nil,
nil,
true, // publicKey is standalone
).(*interpreter.CompositeValue)

publicKey.DeepRemove(inter)
publicKey.DeepRemove(
inter,
true, // publicKey is standalone
)

publicKeyArray2 := publicKey2.GetMember(
inter,
Expand Down
Loading
Loading