[Exec] Use CCF in self-describing mode to encode events (replaces JSON-CDC) #4417
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This uses CCF in fully self-describing mode, so events will encode to about 1/2 the size of JSON-CDC encoding. Using CCF in partially self-describing mode can encode events to 1/14 the size of JSON-CDC but that requires other changes outside of CCF codec.
fxamacker
requested review from
ramtinms,
janezpodhostnik and
pattyshack
as code owners
FVM Benchstat comparisonThis branch with compared with the base branch onflow:master commit 189f62e The command Collapsed results for better readability
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
fxamacker
changed the title
fxamacker
changed the title
Since JSON-CDC encodes less type information than CCF, decoding JSON-CDC and then encoding it in CCF breaks because required type information is missing. Fix this by creating Cadence events and then encoding it in CCF, without involving JSON-CDC.
Codecov Report
@@ Coverage Diff @@
## master #4417 +/- ##
==========================================
+ Coverage 53.71% 54.16% +0.44%
==========================================
Files 888 888
Lines 82612 83737 +1125
==========================================
+ Hits 44378 45352 +974
- Misses 34744 34866 +122
- Partials 3490 3519 +29
Flags with carried forward coverage won't be shown. Click here to find out more.
|
Currently, JSON-CDC decoded event fields are not explicitly sorted and field values are obtained by index in some tests. This isn't compatible with CCF which explicitly sorts fields for determinism. Fix fvm tests by obtaining field values by field identifier to avoid dependency on field order.
Currently, JSON-CDC decoded event fields are not explicitly sorted and field values are obtained by index in some tests. This isn't compatible with CCF which explicitly sorts fields for determinism. Fix fvm tests by obtaining field values by field identifier to avoid dependency on field order.
6 tasks
Currently, JSON-CDC decoded event fields are not explicitly sorted and field values are obtained by index in some tests. This isn't compatible with CCF which explicitly sorts fields for determinism. Fix service events by obtaining field values by field identifier to avoid dependency on field order.
This update gets Cadence PRs 2528 and 2529 to unblock tests.
6 tasks
pattyshack
approved these changes
Jun 5, 2023
| if !ok { | ||
| return nil, invalidCadenceTypeError( | ||
| "counter", | ||
| cdcEvent.Fields[i], |
There was a problem hiding this comment.
just use f instead of cdcEvent.Fields[i] (same elsewhere)
btw, maybe rename f to field (single letter variables are not easily readable / searchable)
There was a problem hiding this comment.
just use f instead of cdcEvent.Fields[i] (same elsewhere)
No, not in this case because f is different from cdcEvent.Fields[i].
f is event type's field (type cadence.Field), while cdcEvent.Fields[i] is event value's field (type cadence.Value).
model/convert/service_event.go
Outdated
|
|
||
| setup := new(flow.EpochSetup) | ||
|
|
||
| setup.Counter = uint64(counter) |
There was a problem hiding this comment.
maybe skip the counter/firstView/finalView temp variable, and just directly assign the values into the setup struct
There was a problem hiding this comment.
Maybe not in this case because we need to first test if counter/firstView/finalView are of specific Cadence types and then cast them as Go types.
| for i, f := range evt.Fields { | ||
| switch f.Identifier { | ||
| case "counter": | ||
| foundFieldCount++ |
There was a problem hiding this comment.
can fields potentially have duplicate identifier (e.g., multiple "counter")? if so, then you should track found field differently
There was a problem hiding this comment.
can fields potentially have duplicate identifier (e.g., multiple "counter")?
No, it would be invalid Cadence event.
fxamacker
force-pushed
the
fxamacker/use-ccf-to-encode-event
branch
from
4995eee to
ec49e0c
Compare
6 tasks
fxamacker
force-pushed
the
fxamacker/use-ccf-to-encode-event
branch
from
3e692fd to
fc5f3c6
Compare
turbolent
reviewed
Jun 6, 2023
koko1123
approved these changes
Jun 6, 2023
turbolent
approved these changes
Jun 6, 2023
|
bors merge |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Updates onflow/cadence#2283
This PR uses CCF in fully self-describing mode, so events will encode to about 1/2 the size of JSON-CDC encoding.
Using CCF in partially self-describing mode can encode events to 1/14 the size of JSON-CDC but that requires other changes outside of CCF codec and other resulting work.
Background
For more info about CCF, see:
TODO
✔️ Requires PR #4390 (merged on June 2, 2023).
✔️ Requires PR onflow/cadence#2528
✔️ Requires PR onflow/cadence#2529
✔️ Requires PR onflow/flow-emulator#408