Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate with atree inlining and data deduplication #2882

Merged

Conversation

fxamacker
Copy link
Member

@fxamacker fxamacker commented Oct 19, 2023

Closes #2809
Updates onflow/atree#292

This commit updates Cadence to use newer version of onflow/atree with register inlining and data deduplication features introduced by onflow/atree#342 and onflow/atree#345.

This PR requires onflow/atree#352.

Changes

  • Updated StorableDecoder callbacks

  • Updated StringAtreeValue to implement atree.ComparableStorable

  • Updated SomeStorable to implement atree.ContainerStorable

  • Updated these structs to implement TypeInfo interface:

    • compositeTypeInfo
    • EmptyTypeInfo
    • VariableSizedStaticType
    • ConstantSizedStaticType
    • DictionaryStaticType
  • Updated decodeStorable() to decode atree inlined array, map, and compact map

  • Updated to use ReadOnly iterators when possible

  • Updated to use NonReadOnly iterators

TODO

Cadence team 🙏 will need to make additional changes:

  • identify and implement more tests (helps prevent data loss)
  • fix metering tests broken by integration (I fixed other tests)
  • resolve some TODOs added as code comments
  • meeting on Oct 20, 2023 to discuss integration, tests, and code review of this PR etc. (thanks for attending and asking great questions! 👍)
  • run Cadence smoke tests
  • @turbolent add *Interpreter and LocationRange to some functions in order to use non-readonly atree.OrderedMap iterators
  • @turbolent check if any storable can contain atree.Array and atree.OrderedMap and needs to implement atree.ContainerStorable interface

  • Targeted PR against master branch
  • Linked to Github issue with discussion and accepted design OR link to spec that describes this work
  • Code follows the standards mentioned here
  • Updated relevant documentation
  • Re-reviewed Files changed in the Github PR explorer
  • Added appropriate labels

Sorry, something went wrong.

This commit updates Cadence to use newer version of onflow/atree
with register inlining and data deduplication features.

- Updated StorableDecoder callbacks

- Updated StringAtreeValue to implement atree.ComparableStorable

- Updated SomeStorable to implement atree.ContainerStorable

- Updated these structs to implement TypeInfo interface:
  * compositeTypeInfo
  * EmptyTypeInfo
  * VariableSizedStaticType
  * ConstantSizedStaticType
  * DictionaryStaticType

- Updated decodeStorable() to decode atree inlined array, map, and
  compact map

- Updated to use ReadOnly iterators when possible

- Updated to use NonReadOnly iterators
@fxamacker fxamacker added Feature Performance Storage Breaking Change Breaks existing stored data (needs a storage migration) labels Oct 19, 2023
@github-actions
Copy link

github-actions bot commented Oct 19, 2023

Cadence Benchstat comparison

This branch with compared with the base branch onflow:feature/atree-register-inlining commit fd29ad8
The command for i in {1..N}; do go test ./... -run=XXX -bench=. -benchmem -shuffle=on; done was used.
Bench tests were run a total of 7 times on each branch.

Collapsed results for better readability

old.txtnew.txt
time/opdelta
CheckContractInterfaceFungibleTokenConformance-483.4µs ± 0%81.8µs ± 0%~(p=1.000 n=1+1)
ContractInterfaceFungibleToken-429.6µs ± 0%27.8µs ± 0%~(p=1.000 n=1+1)
DecodeBatchEventsCCF-4120ms ± 0%118ms ± 0%~(p=1.000 n=1+1)
DecodeBatchEventsJSON-4408ms ± 0%408ms ± 0%~(p=1.000 n=1+1)
DecodeCCF/FlowFees.FeesDeducted-42.45µs ± 0%2.47µs ± 0%~(p=1.000 n=1+1)
DecodeCCF/FlowFees.TokensWithdrawn-42.00µs ± 0%1.97µs ± 0%~(p=1.000 n=1+1)
DecodeCCF/FlowIDTableStaking.DelegatorRewardsPaid-42.54µs ± 0%2.56µs ± 0%~(p=1.000 n=1+1)
DecodeCCF/FlowIDTableStaking.EpochTotalRewardsPaid-42.66µs ± 0%2.69µs ± 0%~(p=1.000 n=1+1)
DecodeCCF/FlowIDTableStaking.NewWeeklyPayout-42.02µs ± 0%2.00µs ± 0%~(p=1.000 n=1+1)
DecodeCCF/FlowIDTableStaking.RewardsPaid-42.33µs ± 0%2.33µs ± 0%~(p=1.000 n=1+1)
DecodeCCF/FlowToken.TokensDeposited-42.40µs ± 0%2.36µs ± 0%~(p=1.000 n=1+1)
DecodeCCF/FlowToken.TokensDeposited_with_nil_receiver-42.33µs ± 0%2.30µs ± 0%~(p=1.000 n=1+1)
DecodeCCF/FlowToken.TokensMinted-42.05µs ± 0%1.98µs ± 0%~(p=1.000 n=1+1)
DecodeCCF/FlowToken.TokensWithdrawn-42.42µs ± 0%2.38µs ± 0%~(p=1.000 n=1+1)
DecodeJSON/FlowFees.FeesDeducted-410.0µs ± 0%9.9µs ± 0%~(p=1.000 n=1+1)
DecodeJSON/FlowFees.TokensWithdrawn-45.79µs ± 0%5.86µs ± 0%~(p=1.000 n=1+1)
DecodeJSON/FlowIDTableStaking.DelegatorRewardsPaid-49.14µs ± 0%9.06µs ± 0%~(p=1.000 n=1+1)
DecodeJSON/FlowIDTableStaking.EpochTotalRewardsPaid-412.4µs ± 0%12.4µs ± 0%~(p=1.000 n=1+1)
DecodeJSON/FlowIDTableStaking.NewWeeklyPayout-45.97µs ± 0%5.92µs ± 0%~(p=1.000 n=1+1)
DecodeJSON/FlowIDTableStaking.RewardsPaid-47.70µs ± 0%7.62µs ± 0%~(p=1.000 n=1+1)
DecodeJSON/FlowToken.TokensDeposited-48.51µs ± 0%8.01µs ± 0%~(p=1.000 n=1+1)
DecodeJSON/FlowToken.TokensDeposited_with_nil_receiver-47.26µs ± 0%7.28µs ± 0%~(p=1.000 n=1+1)
DecodeJSON/FlowToken.TokensMinted-45.94µs ± 0%5.82µs ± 0%~(p=1.000 n=1+1)
DecodeJSON/FlowToken.TokensWithdrawn-48.03µs ± 0%8.00µs ± 0%~(p=1.000 n=1+1)
EncodeBatchEventsCCF-459.8ms ± 0%63.3ms ± 0%~(p=1.000 n=1+1)
EncodeBatchEventsJSON-499.1ms ± 0%99.6ms ± 0%~(p=1.000 n=1+1)
EncodeCCF/FlowFees.FeesDeducted-41.24µs ± 0%1.22µs ± 0%~(p=1.000 n=1+1)
EncodeCCF/FlowFees.TokensWithdrawn-41.00µs ± 0%1.02µs ± 0%~(p=1.000 n=1+1)
EncodeCCF/FlowIDTableStaking.DelegatorRewardsPaid-41.22µs ± 0%1.22µs ± 0%~(p=1.000 n=1+1)
EncodeCCF/FlowIDTableStaking.EpochTotalRewardsPaid-41.36µs ± 0%1.34µs ± 0%~(p=1.000 n=1+1)
EncodeCCF/FlowIDTableStaking.NewWeeklyPayout-41.00µs ± 0%1.00µs ± 0%~(p=1.000 n=1+1)
EncodeCCF/FlowIDTableStaking.RewardsPaid-41.12µs ± 0%1.11µs ± 0%~(p=1.000 n=1+1)
EncodeCCF/FlowToken.TokensDeposited-41.22µs ± 0%1.26µs ± 0%~(p=1.000 n=1+1)
EncodeCCF/FlowToken.TokensDeposited_with_nil_receiver-41.26µs ± 0%1.20µs ± 0%~(p=1.000 n=1+1)
EncodeCCF/FlowToken.TokensMinted-41.00µs ± 0%1.00µs ± 0%~(p=1.000 n=1+1)
EncodeCCF/FlowToken.TokensWithdrawn-41.24µs ± 0%1.22µs ± 0%~(p=1.000 n=1+1)
EncodeJSON/FlowFees.FeesDeducted-42.39µs ± 0%2.39µs ± 0%~(p=1.000 n=1+1)
EncodeJSON/FlowFees.TokensWithdrawn-41.34µs ± 0%1.32µs ± 0%~(p=1.000 n=1+1)
EncodeJSON/FlowIDTableStaking.DelegatorRewardsPaid-42.21µs ± 0%2.14µs ± 0%~(p=1.000 n=1+1)
EncodeJSON/FlowIDTableStaking.EpochTotalRewardsPaid-42.99µs ± 0%2.95µs ± 0%~(p=1.000 n=1+1)
EncodeJSON/FlowIDTableStaking.NewWeeklyPayout-41.35µs ± 0%1.34µs ± 0%~(p=1.000 n=1+1)
EncodeJSON/FlowIDTableStaking.RewardsPaid-41.77µs ± 0%1.76µs ± 0%~(p=1.000 n=1+1)
EncodeJSON/FlowToken.TokensDeposited-42.04µs ± 0%2.06µs ± 0%~(p=1.000 n=1+1)
EncodeJSON/FlowToken.TokensDeposited_with_nil_receiver-41.59µs ± 0%1.57µs ± 0%~(p=1.000 n=1+1)
EncodeJSON/FlowToken.TokensMinted-41.34µs ± 0%1.33µs ± 0%~(p=1.000 n=1+1)
EncodeJSON/FlowToken.TokensWithdrawn-42.01µs ± 0%2.01µs ± 0%~(p=1.000 n=1+1)
ExportType/composite_type-4213ns ± 0%219ns ± 0%~(p=1.000 n=1+1)
ExportType/simple_type-466.3ns ± 0%65.9ns ± 0%~(p=1.000 n=1+1)
InterpretRecursionFib-41.87ms ± 0%1.91ms ± 0%~(p=1.000 n=1+1)
NewInterpreter/new_interpreter-4947ns ± 0%1033ns ± 0%~(p=1.000 n=1+1)
NewInterpreter/new_sub-interpreter-4469ns ± 0%464ns ± 0%~(p=1.000 n=1+1)
ParseArray-46.12ms ± 0%6.06ms ± 0%~(p=1.000 n=1+1)
ParseDeploy/byte_array-49.86ms ± 0%8.98ms ± 0%~(p=1.000 n=1+1)
ParseDeploy/decode_hex-41.01ms ± 0%1.02ms ± 0%~(p=1.000 n=1+1)
ParseFungibleToken/With_memory_metering-4147µs ± 0%143µs ± 0%~(p=1.000 n=1+1)
ParseFungibleToken/Without_memory_metering-4114µs ± 0%113µs ± 0%~(p=1.000 n=1+1)
ParseInfix-45.29µs ± 0%5.44µs ± 0%~(p=1.000 n=1+1)
QualifiedIdentifierCreation/One_level-42.17ns ± 0%2.17ns ± 0%~(p=1.000 n=1+1)
QualifiedIdentifierCreation/Three_levels-483.9ns ± 0%84.1ns ± 0%~(p=1.000 n=1+1)
RuntimeResourceDictionaryValues-44.07ms ± 0%1.86ms ± 0%~(p=1.000 n=1+1)
RuntimeScriptNoop-43.53µs ± 0%3.58µs ± 0%~(p=1.000 n=1+1)
SuperTypeInference/arrays-4237ns ± 0%238ns ± 0%~(p=1.000 n=1+1)
SuperTypeInference/composites-489.0ns ± 0%94.9ns ± 0%~(p=1.000 n=1+1)
SuperTypeInference/integers-4130ns ± 0%130ns ± 0%~(p=1.000 n=1+1)
ValueIsSubtypeOfSemaType-468.3ns ± 0%76.8ns ± 0%~(p=1.000 n=1+1)
 
alloc/opdelta
CheckContractInterfaceFungibleTokenConformance-449.3kB ± 0%49.2kB ± 0%~(p=1.000 n=1+1)
ContractInterfaceFungibleToken-423.4kB ± 0%23.4kB ± 0%~(all equal)
DecodeBatchEventsCCF-467.4MB ± 0%67.4MB ± 0%~(p=1.000 n=1+1)
DecodeBatchEventsJSON-4246MB ± 0%246MB ± 0%~(p=1.000 n=1+1)
DecodeCCF/FlowFees.FeesDeducted-41.41kB ± 0%1.41kB ± 0%~(all equal)
DecodeCCF/FlowFees.TokensWithdrawn-41.22kB ± 0%1.22kB ± 0%~(all equal)
DecodeCCF/FlowIDTableStaking.DelegatorRewardsPaid-41.49kB ± 0%1.49kB ± 0%~(all equal)
DecodeCCF/FlowIDTableStaking.EpochTotalRewardsPaid-41.50kB ± 0%1.50kB ± 0%~(all equal)
DecodeCCF/FlowIDTableStaking.NewWeeklyPayout-41.26kB ± 0%1.26kB ± 0%~(all equal)
DecodeCCF/FlowIDTableStaking.RewardsPaid-41.38kB ± 0%1.38kB ± 0%~(all equal)
DecodeCCF/FlowToken.TokensDeposited-41.34kB ± 0%1.34kB ± 0%~(all equal)
DecodeCCF/FlowToken.TokensDeposited_with_nil_receiver-41.33kB ± 0%1.33kB ± 0%~(all equal)
DecodeCCF/FlowToken.TokensMinted-41.22kB ± 0%1.22kB ± 0%~(all equal)
DecodeCCF/FlowToken.TokensWithdrawn-41.35kB ± 0%1.35kB ± 0%~(all equal)
DecodeJSON/FlowFees.FeesDeducted-46.02kB ± 0%6.02kB ± 0%~(all equal)
DecodeJSON/FlowFees.TokensWithdrawn-43.62kB ± 0%3.62kB ± 0%~(all equal)
DecodeJSON/FlowIDTableStaking.DelegatorRewardsPaid-45.45kB ± 0%5.45kB ± 0%~(all equal)
DecodeJSON/FlowIDTableStaking.EpochTotalRewardsPaid-47.37kB ± 0%7.37kB ± 0%~(all equal)
DecodeJSON/FlowIDTableStaking.NewWeeklyPayout-43.66kB ± 0%3.66kB ± 0%~(all equal)
DecodeJSON/FlowIDTableStaking.RewardsPaid-44.55kB ± 0%4.55kB ± 0%~(all equal)
DecodeJSON/FlowToken.TokensDeposited-44.91kB ± 0%4.91kB ± 0%~(all equal)
DecodeJSON/FlowToken.TokensDeposited_with_nil_receiver-44.49kB ± 0%4.49kB ± 0%~(all equal)
DecodeJSON/FlowToken.TokensMinted-43.62kB ± 0%3.62kB ± 0%~(all equal)
DecodeJSON/FlowToken.TokensWithdrawn-44.91kB ± 0%4.91kB ± 0%~(all equal)
EncodeBatchEventsCCF-437.1MB ± 0%37.1MB ± 0%~(p=1.000 n=1+1)
EncodeBatchEventsJSON-434.0MB ± 0%34.0MB ± 0%~(p=1.000 n=1+1)
EncodeCCF/FlowFees.FeesDeducted-4736B ± 0%736B ± 0%~(all equal)
EncodeCCF/FlowFees.TokensWithdrawn-4688B ± 0%688B ± 0%~(all equal)
EncodeCCF/FlowIDTableStaking.DelegatorRewardsPaid-4800B ± 0%800B ± 0%~(all equal)
EncodeCCF/FlowIDTableStaking.EpochTotalRewardsPaid-4768B ± 0%768B ± 0%~(all equal)
EncodeCCF/FlowIDTableStaking.NewWeeklyPayout-4704B ± 0%704B ± 0%~(all equal)
EncodeCCF/FlowIDTableStaking.RewardsPaid-4784B ± 0%784B ± 0%~(all equal)
EncodeCCF/FlowToken.TokensDeposited-4752B ± 0%752B ± 0%~(all equal)
EncodeCCF/FlowToken.TokensDeposited_with_nil_receiver-4736B ± 0%736B ± 0%~(all equal)
EncodeCCF/FlowToken.TokensMinted-4688B ± 0%688B ± 0%~(all equal)
EncodeCCF/FlowToken.TokensWithdrawn-4752B ± 0%752B ± 0%~(all equal)
EncodeJSON/FlowFees.FeesDeducted-4768B ± 0%768B ± 0%~(all equal)
EncodeJSON/FlowFees.TokensWithdrawn-4408B ± 0%408B ± 0%~(all equal)
EncodeJSON/FlowIDTableStaking.DelegatorRewardsPaid-4760B ± 0%760B ± 0%~(all equal)
EncodeJSON/FlowIDTableStaking.EpochTotalRewardsPaid-4952B ± 0%952B ± 0%~(all equal)
EncodeJSON/FlowIDTableStaking.NewWeeklyPayout-4424B ± 0%424B ± 0%~(all equal)
EncodeJSON/FlowIDTableStaking.RewardsPaid-4624B ± 0%624B ± 0%~(all equal)
EncodeJSON/FlowToken.TokensDeposited-4680B ± 0%680B ± 0%~(all equal)
EncodeJSON/FlowToken.TokensDeposited_with_nil_receiver-4544B ± 0%544B ± 0%~(all equal)
EncodeJSON/FlowToken.TokensMinted-4416B ± 0%416B ± 0%~(all equal)
EncodeJSON/FlowToken.TokensWithdrawn-4672B ± 0%672B ± 0%~(all equal)
ExportType/composite_type-4136B ± 0%136B ± 0%~(all equal)
ExportType/simple_type-40.00B 0.00B ~(all equal)
InterpretRecursionFib-41.00MB ± 0%1.00MB ± 0%~(all equal)
NewInterpreter/new_interpreter-4976B ± 0%976B ± 0%~(all equal)
NewInterpreter/new_sub-interpreter-4200B ± 0%200B ± 0%~(all equal)
ParseArray-42.65MB ± 0%2.65MB ± 0%~(p=1.000 n=1+1)
ParseDeploy/byte_array-44.09MB ± 0%4.09MB ± 0%~(p=1.000 n=1+1)
ParseDeploy/decode_hex-4214kB ± 0%214kB ± 0%~(p=1.000 n=1+1)
ParseFungibleToken/With_memory_metering-428.9kB ± 0%28.9kB ± 0%~(p=1.000 n=1+1)
ParseFungibleToken/Without_memory_metering-428.9kB ± 0%28.9kB ± 0%~(p=1.000 n=1+1)
ParseInfix-41.92kB ± 0%1.92kB ± 0%~(all equal)
QualifiedIdentifierCreation/One_level-40.00B 0.00B ~(all equal)
QualifiedIdentifierCreation/Three_levels-464.0B ± 0%64.0B ± 0%~(all equal)
RuntimeResourceDictionaryValues-42.29MB ± 0%1.26MB ± 0%~(p=1.000 n=1+1)
RuntimeScriptNoop-43.02kB ± 0%3.02kB ± 0%~(all equal)
SuperTypeInference/arrays-496.0B ± 0%96.0B ± 0%~(all equal)
SuperTypeInference/composites-40.00B 0.00B ~(all equal)
SuperTypeInference/integers-40.00B 0.00B ~(all equal)
ValueIsSubtypeOfSemaType-448.0B ± 0%48.0B ± 0%~(all equal)
 
allocs/opdelta
CheckContractInterfaceFungibleTokenConformance-4811 ± 0%811 ± 0%~(all equal)
ContractInterfaceFungibleToken-4370 ± 0%370 ± 0%~(all equal)
DecodeBatchEventsCCF-41.48M ± 0%1.48M ± 0%~(p=1.000 n=1+1)
DecodeBatchEventsJSON-44.70M ± 0%4.70M ± 0%~(p=1.000 n=1+1)
DecodeCCF/FlowFees.FeesDeducted-430.0 ± 0%30.0 ± 0%~(all equal)
DecodeCCF/FlowFees.TokensWithdrawn-426.0 ± 0%26.0 ± 0%~(all equal)
DecodeCCF/FlowIDTableStaking.DelegatorRewardsPaid-430.0 ± 0%30.0 ± 0%~(all equal)
DecodeCCF/FlowIDTableStaking.EpochTotalRewardsPaid-432.0 ± 0%32.0 ± 0%~(all equal)
DecodeCCF/FlowIDTableStaking.NewWeeklyPayout-426.0 ± 0%26.0 ± 0%~(all equal)
DecodeCCF/FlowIDTableStaking.RewardsPaid-429.0 ± 0%29.0 ± 0%~(all equal)
DecodeCCF/FlowToken.TokensDeposited-431.0 ± 0%31.0 ± 0%~(all equal)
DecodeCCF/FlowToken.TokensDeposited_with_nil_receiver-429.0 ± 0%29.0 ± 0%~(all equal)
DecodeCCF/FlowToken.TokensMinted-426.0 ± 0%26.0 ± 0%~(all equal)
DecodeCCF/FlowToken.TokensWithdrawn-431.0 ± 0%31.0 ± 0%~(all equal)
DecodeJSON/FlowFees.FeesDeducted-4126 ± 0%126 ± 0%~(all equal)
DecodeJSON/FlowFees.TokensWithdrawn-471.0 ± 0%71.0 ± 0%~(all equal)
DecodeJSON/FlowIDTableStaking.DelegatorRewardsPaid-4102 ± 0%102 ± 0%~(all equal)
DecodeJSON/FlowIDTableStaking.EpochTotalRewardsPaid-4159 ± 0%159 ± 0%~(all equal)
DecodeJSON/FlowIDTableStaking.NewWeeklyPayout-470.0 ± 0%70.0 ± 0%~(all equal)
DecodeJSON/FlowIDTableStaking.RewardsPaid-487.0 ± 0%87.0 ± 0%~(all equal)
DecodeJSON/FlowToken.TokensDeposited-495.0 ± 0%95.0 ± 0%~(all equal)
DecodeJSON/FlowToken.TokensDeposited_with_nil_receiver-486.0 ± 0%86.0 ± 0%~(all equal)
DecodeJSON/FlowToken.TokensMinted-471.0 ± 0%71.0 ± 0%~(all equal)
DecodeJSON/FlowToken.TokensWithdrawn-495.0 ± 0%95.0 ± 0%~(all equal)
EncodeBatchEventsCCF-4467k ± 0%467k ± 0%~(p=1.000 n=1+1)
EncodeBatchEventsJSON-4757k ± 0%757k ± 0%~(p=1.000 n=1+1)
EncodeCCF/FlowFees.FeesDeducted-49.00 ± 0%9.00 ± 0%~(all equal)
EncodeCCF/FlowFees.TokensWithdrawn-49.00 ± 0%9.00 ± 0%~(all equal)
EncodeCCF/FlowIDTableStaking.DelegatorRewardsPaid-49.00 ± 0%9.00 ± 0%~(all equal)
EncodeCCF/FlowIDTableStaking.EpochTotalRewardsPaid-49.00 ± 0%9.00 ± 0%~(all equal)
EncodeCCF/FlowIDTableStaking.NewWeeklyPayout-49.00 ± 0%9.00 ± 0%~(all equal)
EncodeCCF/FlowIDTableStaking.RewardsPaid-49.00 ± 0%9.00 ± 0%~(all equal)
EncodeCCF/FlowToken.TokensDeposited-410.0 ± 0%10.0 ± 0%~(all equal)
EncodeCCF/FlowToken.TokensDeposited_with_nil_receiver-410.0 ± 0%10.0 ± 0%~(all equal)
EncodeCCF/FlowToken.TokensMinted-49.00 ± 0%9.00 ± 0%~(all equal)
EncodeCCF/FlowToken.TokensWithdrawn-410.0 ± 0%10.0 ± 0%~(all equal)
EncodeJSON/FlowFees.FeesDeducted-417.0 ± 0%17.0 ± 0%~(all equal)
EncodeJSON/FlowFees.TokensWithdrawn-410.0 ± 0%10.0 ± 0%~(all equal)
EncodeJSON/FlowIDTableStaking.DelegatorRewardsPaid-414.0 ± 0%14.0 ± 0%~(all equal)
EncodeJSON/FlowIDTableStaking.EpochTotalRewardsPaid-423.0 ± 0%23.0 ± 0%~(all equal)
EncodeJSON/FlowIDTableStaking.NewWeeklyPayout-410.0 ± 0%10.0 ± 0%~(all equal)
EncodeJSON/FlowIDTableStaking.RewardsPaid-413.0 ± 0%13.0 ± 0%~(all equal)
EncodeJSON/FlowToken.TokensDeposited-417.0 ± 0%17.0 ± 0%~(all equal)
EncodeJSON/FlowToken.TokensDeposited_with_nil_receiver-412.0 ± 0%12.0 ± 0%~(all equal)
EncodeJSON/FlowToken.TokensMinted-411.0 ± 0%11.0 ± 0%~(all equal)
EncodeJSON/FlowToken.TokensWithdrawn-416.0 ± 0%16.0 ± 0%~(all equal)
ExportType/composite_type-43.00 ± 0%3.00 ± 0%~(all equal)
ExportType/simple_type-40.00 0.00 ~(all equal)
InterpretRecursionFib-418.9k ± 0%18.9k ± 0%~(all equal)
NewInterpreter/new_interpreter-416.0 ± 0%16.0 ± 0%~(all equal)
NewInterpreter/new_sub-interpreter-44.00 ± 0%4.00 ± 0%~(all equal)
ParseArray-459.6k ± 0%59.6k ± 0%~(p=1.000 n=1+1)
ParseDeploy/byte_array-489.4k ± 0%89.4k ± 0%~(p=1.000 n=1+1)
ParseDeploy/decode_hex-463.0 ± 0%63.0 ± 0%~(all equal)
ParseFungibleToken/With_memory_metering-4768 ± 0%768 ± 0%~(all equal)
ParseFungibleToken/Without_memory_metering-4768 ± 0%768 ± 0%~(all equal)
ParseInfix-448.0 ± 0%48.0 ± 0%~(all equal)
QualifiedIdentifierCreation/One_level-40.00 0.00 ~(all equal)
QualifiedIdentifierCreation/Three_levels-42.00 ± 0%2.00 ± 0%~(all equal)
RuntimeResourceDictionaryValues-437.0k ± 0%20.6k ± 0%~(p=1.000 n=1+1)
RuntimeScriptNoop-450.0 ± 0%50.0 ± 0%~(all equal)
SuperTypeInference/arrays-43.00 ± 0%3.00 ± 0%~(all equal)
SuperTypeInference/composites-40.00 0.00 ~(all equal)
SuperTypeInference/integers-40.00 0.00 ~(all equal)
ValueIsSubtypeOfSemaType-41.00 ± 0%1.00 ± 0%~(all equal)
 

@fxamacker fxamacker force-pushed the fxamacker/integrate-with-atree branch from c8cc585 to ca1a9ea Compare October 19, 2023 02:21
Copy link
Member

@turbolent turbolent left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work!

runtime/cmd/decode-state-values/main.go Outdated Show resolved Hide resolved
runtime/cmd/decode-state-values/main.go Outdated Show resolved Hide resolved
runtime/interpreter/encode.go Outdated Show resolved Hide resolved
runtime/interpreter/encode.go Outdated Show resolved Hide resolved
runtime/interpreter/encode.go Outdated Show resolved Hide resolved
runtime/interpreter/value.go Outdated Show resolved Hide resolved
runtime/interpreter/value.go Show resolved Hide resolved
@@ -18043,7 +18059,7 @@ func (v *DictionaryValue) ForEachKey(
}

iterate := func() {
err := v.dictionary.IterateKeys(
err := v.dictionary.IterateReadOnlyKeys(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we sure that the procedure passed to DictionaryValue.ForEachKey does not mutate? I'd be conservative and switch to a read-only iterator after we ensured it. It might also be good to indicate that in the name then

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we sure that the procedure passed to DictionaryValue.ForEachKey does not mutate?

Hmm, do you mean the procedure might mutate map value by iterated key?

Using a non-readonly vs readonly iterator probably wouldn't make a difference here because mutation through iterator is done by callback in returned map value. Since procedure only processes map key, mutating map value is not possible by callback. Actually, atree only has readonly iterators for map keys for the same reason.

Thoughts?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[...] because mutation through iterator is done by callback in returned map value

Not quite sure what that means. Do you maybe have an example?

mutating map value is not possible by callback

The callback is free to do anything, including removing keys, inserting new keys, mutating values in the dictionary, etc.

I guess your point is still that mutable vs read-only iterator doesn't matter?

runtime/interpreter/value.go Show resolved Hide resolved
@@ -28,6 +28,7 @@ type StringAtreeValue string

var _ atree.Value = StringAtreeValue("")
var _ atree.Storable = StringAtreeValue("")
var _ atree.ComparableStorable = StringAtreeValue("")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this also needed for Uint64AtreeValue?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unless I'm mistaken, it probably isn't needed because:

  • atree.ComparableStorable is only needed for atree.OrderedMap keys in Cadence CompositeValue
  • StringAtreeValue seems to be the only key type used by CompositeValue

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

StringAtreeValue is also used for "non-interpreter.Value" storage, i.e. for the atree ordered maps used for account storage, and Uint64AtreeValue is only used for those.

For example, saving a value to /storage/foo actually is implemented as storing a value inside of the account storage's ordered map for the storage domain, under key StringAtreeValue("foo")

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was discussed in today's meeting and it isn't for Uint64AtreeValue needed as mentioned above.

Co-authored-by: Bastian Müller <bastian@turbolent.com>
@fxamacker fxamacker force-pushed the fxamacker/integrate-with-atree branch from af06ef7 to bc7b622 Compare October 19, 2023 15:26
Co-authored-by: Bastian Müller <bastian@turbolent.com>
runtime/cmd/decode-state-values/main.go Outdated Show resolved Hide resolved
runtime/interpreter/statictype.go Outdated Show resolved Hide resolved
runtime/interpreter/statictype.go Outdated Show resolved Hide resolved
fxamacker and others added 3 commits October 19, 2023 15:31

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
Co-authored-by: Supun Setunga <supun.setunga@gmail.com>
@codecov
Copy link

codecov bot commented Oct 19, 2023

Codecov Report

Attention: 138 lines in your changes are missing coverage. Please review.

Comparison is base (c74379d) 79.51% compared to head (ea8fd35) 79.52%.
Report is 1 commits behind head on feature/atree-register-inlining.

Files Patch % Lines
runtime/interpreter/decode.go 56.56% 38 Missing and 5 partials ⚠️
runtime/interpreter/value.go 89.24% 40 Missing ⚠️
runtime/interpreter/storagemapkey.go 0.00% 20 Missing ⚠️
runtime/interpreter/encode.go 66.66% 10 Missing and 2 partials ⚠️
runtime/convertValues.go 81.81% 4 Missing and 2 partials ⚠️
runtime/interpreter/statictype.go 80.00% 6 Missing ⚠️
runtime/interpreter/value_function.go 0.00% 3 Missing ⚠️
runtime/interpreter/storage.go 66.66% 1 Missing and 1 partial ⚠️
...e/interpreter/value_accountcapabilitycontroller.go 33.33% 2 Missing ⚠️
...e/interpreter/value_storagecapabilitycontroller.go 33.33% 2 Missing ⚠️
... and 2 more
Additional details and impacted files
@@                         Coverage Diff                         @@
##           feature/atree-register-inlining    #2882      +/-   ##
===================================================================
+ Coverage                            79.51%   79.52%   +0.01%     
===================================================================
  Files                                  336      336              
  Lines                                79191    79560     +369     
===================================================================
+ Hits                                 62969    63273     +304     
- Misses                               13916    13974      +58     
- Partials                              2306     2313       +7     
Flag Coverage Δ
unittests 79.52% <79.15%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…ows using read-only iterator
…tion in tests
…eturn shallow copy
@fxamacker
Copy link
Member Author

@turbolent Thanks for reviewing commit 1ecf35c on Friday!

Today is Canadian holiday and Friday was US holiday, maybe we can sync about this before more holidays come up. 😄

I had a look through 1ecf35c and it looks like there several cases where atRoot should be true. Not sure if those cases were intentionally false. I think I caught all of them, but maybe double check too

Unless I missed something, I think we can skip incorporating all of the separate suggestions related to the last feedback. However, my reasons vary for some suggestions and I would like to get your thoughts about some new ideas/questions. 🙏

I will summarize here instead of replying to each suggestion individually because they are related.

There are two main scenarios in your comments:

  • Transferred value from Iterators, Get, or ReadStored, with remove parameter as false (in interpreter.go, value.go)
  • Transferred value from ReadStored with remove parameter as true (in account.go)

First Scenario

atRoot is false because:

  • Transferred value is from Iterators, Get() or ReadStored(). So the transferred value isn't a root value since it is an element read from parent container.
  • atRoot parameter only takes effect if remove parameter is true in Transfer(). In other words, in Transfer(), storage check is performed if both atRoot and remove parameters are true. Since remove is false in this scenario, atRoot is effectively no-op. This behavior is exactly same as before this commit (without atRoot).

if remove {
err = v.array.PopIterate(func(storable atree.Storable) {
interpreter.RemoveReferencedSlab(storable)
})
if err != nil {
panic(errors.NewExternalError(err))
}
interpreter.maybeValidateAtreeValue(v.array)
if atRoot {
interpreter.maybeValidateAtreeStorage()
}
interpreter.RemoveReferencedSlab(storable)
}

Second Scenario

This is more involved because:

  • Transferred value is read from storage map (so value isn't root because it has parent)
  • remove is true for Transfer(), so it is possible to remove storable for this value from storage while storage map still manages it officially
  • after Transfer(), we set nil in its place in storage map

readValue := inter.ReadStored(provider, InboxStorageDomain, storageMapKey)
if readValue == nil {
return interpreter.Nil
}
publishedValue := readValue.(*interpreter.PublishedValue)
if !ok {
panic(errors.NewUnreachableError())
}
typeParameterPair := invocation.TypeParameterTypes.Oldest()
if typeParameterPair == nil {
panic(errors.NewUnreachableError())
}
ty := sema.NewCapabilityType(gauge, typeParameterPair.Value)
publishedType := publishedValue.Value.StaticType(invocation.Interpreter)
if !inter.IsSubTypeOfSemaType(publishedType, ty) {
panic(interpreter.ForceCastTypeMismatchError{
ExpectedType: ty,
ActualType: inter.MustConvertStaticToSemaType(publishedType),
LocationRange: locationRange,
})
}
value := publishedValue.Value.Transfer(
inter,
locationRange,
atree.Address{},
true,
nil,
nil,
false,
)
inter.WriteStored(
provider,
InboxStorageDomain,
storageMapKey,
nil,
)

Alternative Suggestion for Second Scenario

Would it make sense to do these instead?

  • remove data from storage map (not storage)
  • transfer removed data to stack address and remove data in owner address in storage

So atRoot in the second scenario really depends on the flow of data outlined above. For example, if transferred value is removed from storage map first, atRoot is true for transferred value; othewise, I think atRoot should be false because the transferred data has parent (not root).

Thoughts?

@turbolent
Copy link
Member

turbolent commented Nov 21, 2023

TODO for this PR:

  • Rename the atRoot parameter for the Transfer and DeepRemove functions and clarify it in a comment @fxamacker
  • Double-check and potentially switch read-only iterators in Transfer functions to write-iterator, as the potential nested removal (remove == true) is a mutation @fxamacker
  • Adjust argument for DeepRemove calls to align with semantics of atRoot parameter of Transfer @turbolent

TODO for another PR, ideally before:

  • Simplify Transfer function @turbolent
    • Can we maybe split the function into one for value-kinded (e.g. Copy) and one for resource-kinded transfer (e.g. Move)?
    • Can the remove flag be removed and inferred? Maybe from resource-kindedness?
    • When constructing physical a new value ("physical copy" in the target address), do not remove while iterating.
      Instead, create the copy, then DeepRemove (if needed)

If remove parameter is true in Transfer(), iterated values are
removed so non-readonly iterators need to be used.
- Renamed atRoot param to hasNoParentContainer for Transfer() and
  DeepRemove()
- Added comments for each instances of hasNoParentContainer for
  Transfer()
- For SomeValue.Transfer() and PublishedValue.Transfer(), use
  received hasNoParentContainer argument as inner value's
  Transfer() parameter
For SomeValue.DeepRemove(), PathCapabilityValue.Transfer(),
and IDCapabilityValue.DeepRemove(), use received
hasNoParentContainer argument as inner value's DeepRemove()
parameter.

Also, add comments for hasNoParentContainer in DeepRemove().
@fxamacker
Copy link
Member Author

fxamacker commented Nov 22, 2023

Hey @turbolent, thanks for a great meeting this morning to go over this PR again and also cover other issues in DeepRemove() and Transfer().

TODO for this PR:

Thanks for posting these TODOs from our meeting! 👍

  • Rename the atRoot parameter for the Transfer and DeepRemove functions and clarify it in a comment @fxamacker

Done in commit: Rename atRoot param for Transfer() and DeepRemove()

  • Double-check and potentially switch read-only iterators in Transfer functions to write-iterator, as the potential nested removal (remove == true) is a mutation @fxamacker

Done in commit: Use Atree non-readonly iterators in Transfer()

  • Adjust argument for DeepRemove calls to align with semantics of atRoot parameter of Transfer @turbolent

Done in commit: Use hasNoParentContainer for inner value DeepRemove()

I hope you don't mind, 🙏 I was already there and wanted to have this PR wrapped up.

TODO for another PR, ideally before:

  • Simplify Transfer function @turbolent
  • Can we maybe split the function into one for value-kinded (e.g. Copy) and one for resource-kinded transfer (e.g. Move)?
  • Can the remove flag be removed and inferred? Maybe from resource-kindedness?
  • When constructing physical a new value ("physical copy" in the target address), do not remove while iterating.
    Instead, create the copy, then DeepRemove (if needed)

Thanks for tackling those outside this PR! 👍

Comment on lines +688 to +690
if nestedLevels == 1 {
return s.encode(e)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can this also be handled as part of encodeMultipleNestedLevels (instead of returning error there) ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for delay in reply, I missed this comment. 🙏

Maybe not in this case because encoding format is differently for nestedLevel == 1 (using encode()) vs nestedLevel > 1 (using encodeMultipleNestedLevels()):

  • if nested level == 1, only innermost unwrapped storable is encoded
  • if nested level > 1, then 2-element array is encoded: first element is nested levels, second element is innermost unwrapped storable.

Copy link
Contributor

@ramtinms ramtinms left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me

@turbolent
Copy link
Member

The latest commits look good 👍

Copy link
Member

@turbolent turbolent left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Double checked the switches of iterators from mutating to read-only ones, LGTM in that regard 👍

valueComparator := newValueComparator(interpreter, locationRange)
hashInputProvider := newHashInputProvider(interpreter, locationRange)

iterator, err := v.dictionary.Iterator(valueComparator, hashInputProvider)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have to use a mutable iterator here, because the values that are returned by the iterator might get mutated, so need to have the parent notification callback set up, correct?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct. 👍

@turbolent
Copy link
Member

@fxamacker Do you happen to remember what I meant with

When constructing physical a new value ("physical copy" in the target address), do not remove while iterating.
Instead, create the copy, then DeepRemove (if needed)

I vaguely remember we discussed how there's a potential problem with that approach in the inlining version

@fxamacker
Copy link
Member Author

@fxamacker Do you happen to remember what I meant with

When constructing physical a new value ("physical copy" in the target address), do not remove while iterating.
Instead, create the copy, then DeepRemove (if needed)

I vaguely remember we discussed how there's a potential problem with that approach in the inlining version

@turbolent Yeah, I remember that too. For context:

  • we added parameter to Transfer(), etc. to support this PR, which increased complexity of Transfer()
  • we reviewed Transfer() together and found old code in `Transfer() could use refactor (see below) but that refactor is outside scope of this PR

Part of the discussion to simplify Transfer():

  • it was complicated already because both "copy" and "move" functionality is combined in one function,
  • we had to add new parameter hasNoParentContainer to both Transfer() and DeepRemove() to make atree storage health check work (uncovered by cadence smoke test)
  • since atree storage health fail during removing while iterating nested structures in Transfer(), we want to simplify by separating copy and remove in Transfer.

Copy link
Member

@turbolent turbolent left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing work @fxamacker! 👏 👏 👏

@fxamacker fxamacker enabled auto-merge January 24, 2024 20:51

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
…te-with-atree
@turbolent turbolent disabled auto-merge January 24, 2024 21:01
@turbolent turbolent merged commit deffd50 into feature/atree-register-inlining Jan 24, 2024
8 of 11 checks passed
@turbolent turbolent deleted the fxamacker/integrate-with-atree branch January 24, 2024 21:02
turbolent pushed a commit that referenced this pull request Jan 26, 2024
Cadence smoke test is reporting false alarm about
"slab was not reachable".

To summarize, storage.CheckHealth() should be called after
array.DeepRemove(), rather than in the middle of array.DeepRemove().

CheckHealth() is called in the middle of array.DeepRemove() when:
- array.DeepRemove() calls childArray1 and childArray2 DeepRemove()
- DeepRemove() calls maybeValidateAtreeValue()
- maybeValidateAtreeValue() calls CheckHealth()

The details are more complex than this oversimplified summary.

For more context, see comments/discussion on GitHub at
#2882 (comment)
turbolent pushed a commit that referenced this pull request Jan 26, 2024
Cadence smoke test reported another false alarm about
"slab was not reachable".

To summarize, storage.CheckHealth() should be called after
DictionaryValue.Transfer() with remove flag, rather than
in the middle of DictionaryValue.Transfer() with remove flag.

The details are more complex than this oversimplified summary.

For more context, see GitHub comments at
#2882 (comment)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature Performance Storage Breaking Change Breaks existing stored data (needs a storage migration)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants