Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chunk Data Model supports per-chunk service event mapping #6744

Merged
merged 33 commits into from
Dec 11, 2024
Merged
Changes from 1 commit
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
e59dc76
add chunk model changes and encoding tests
jordanschalm Nov 20, 2024
869feff
add servcie event indices to chunk
jordanschalm Nov 20, 2024
6e4c8de
revert test package change
jordanschalm Nov 20, 2024
d383068
add test file
jordanschalm Nov 20, 2024
97515e8
rm indices from fixture
jordanschalm Nov 20, 2024
ec7c74e
add ServiceEventsByChunk method on result
jordanschalm Nov 20, 2024
e906251
fix chunk verifier test
jordanschalm Nov 20, 2024
b9849c6
add indices getter tests
jordanschalm Nov 20, 2024
8961e51
add ServiceEventsByChunk tests
jordanschalm Nov 21, 2024
5579f2a
chunk verifier tests
jordanschalm Nov 21, 2024
f1125b3
improve docs
jordanschalm Nov 21, 2024
718a5ee
improve docs
jordanschalm Nov 21, 2024
44f3c1a
adjust field and documentation
jordanschalm Nov 28, 2024
13aba21
update tests
jordanschalm Nov 28, 2024
defcdea
wip
jordanschalm Nov 29, 2024
405626e
update BlockResult tests
jordanschalm Dec 2, 2024
447017b
fix test name
jordanschalm Dec 2, 2024
e465bbc
fix chunk verifier test
jordanschalm Dec 2, 2024
140b324
backward compatible RLP encoding + tests
jordanschalm Dec 3, 2024
49bcbf6
note rpc change requirements
jordanschalm Dec 3, 2024
176100f
Merge branch 'feature/efm-recovery' into jord/6622-chunk-service-events
jordanschalm Dec 3, 2024
ee99871
Apply suggestions from code review
jordanschalm Dec 9, 2024
4d40dcf
add deprecated notes
jordanschalm Dec 9, 2024
9b7735e
Merge branch 'jord/6622-chunk-service-events' of github.com:onflow/fl…
jordanschalm Dec 9, 2024
ad346e8
RLP notes
jordanschalm Dec 9, 2024
b484203
rename var
jordanschalm Dec 9, 2024
a61e30e
add test case demonstrating rlp order dependence
jordanschalm Dec 9, 2024
6fe51f6
EncodeDecodeDifferentVersions docs
jordanschalm Dec 9, 2024
4a651b6
sanity check for service event count field
jordanschalm Dec 10, 2024
4a27b73
fix ER test
jordanschalm Dec 10, 2024
e1ee2f0
add context to chunkverifier test
jordanschalm Dec 10, 2024
9c1ef70
refactor generateEvents
jordanschalm Dec 10, 2024
9a10320
Update model/flow/chunk.go
jordanschalm Dec 10, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
add chunk model changes and encoding tests
  • Loading branch information
jordanschalm committed Nov 20, 2024

Verified

This commit was signed with the committer’s verified signature.
jordanschalm Jordan Schalm
commit e59dc76e4d2793ba43113b356dc0ef972f88569f
39 changes: 38 additions & 1 deletion model/flow/chunk.go
Original file line number Diff line number Diff line change
@@ -6,6 +6,8 @@ import (

"github.com/ipfs/go-cid"
"github.com/vmihailenco/msgpack/v4"

"github.com/onflow/flow-go/model/encoding/rlp"
)

var EmptyEventCollectionID Identifier
@@ -20,19 +22,54 @@ func init() {
}
}

// TODO doc
type chunkBodyV0 struct {
CollectionIndex uint
StartState StateCommitment
EventCollection Identifier
BlockID Identifier
TotalComputationUsed uint64
NumberOfTransactions uint64
}

type ChunkBody struct {
CollectionIndex uint

// execution info
StartState StateCommitment // start state when starting executing this chunk
EventCollection Identifier // Events generated by executing results
BlockID Identifier // Block id of the execution result this chunk belongs to
// ServiceEventIndices is a list of indices of service events which were emitted.
// If ServiceEventIndices is nil, this indicates that this chunk was created by an older software version
// which did support specifying a mapping between chunks and service events.
// TODO doc
ServiceEventIndices []uint32
BlockID Identifier // Block id of the execution result this chunk belongs to

// Computation consumption info
TotalComputationUsed uint64 // total amount of computation used by running all txs in this chunk
NumberOfTransactions uint64 // number of transactions inside the collection
}

jordanschalm marked this conversation as resolved.
Show resolved Hide resolved
// Fingerprint returns the unique binary representation for the receiver ChunkBody,
// used to compute the ID (hash).
// The fingerprint is backward-compatible with the prior data model for ChunkBody: chunkBodyV0.
// - All new ChunkBody instances must have non-nil ServiceEventIndices
// - A nil ServiceEventIndices field indicates a v0 version of ChunkBody
// - when computing the ID of such a ChunkBody, the ServiceEventIndices field is omitted from the fingerprint
func (ch ChunkBody) Fingerprint() []byte {
if ch.ServiceEventIndices == nil {
return rlp.NewMarshaler().MustMarshal(chunkBodyV0{
CollectionIndex: ch.CollectionIndex,
StartState: ch.StartState,
EventCollection: ch.EventCollection,
BlockID: ch.BlockID,
TotalComputationUsed: ch.TotalComputationUsed,
NumberOfTransactions: ch.NumberOfTransactions,
})
}
return rlp.NewMarshaler().MustMarshal(ch)
}

type Chunk struct {
ChunkBody

157 changes: 145 additions & 12 deletions model/flow/chunk_test.go
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@
package flow_test
package flow

import (
"encoding/json"
"testing"

"github.com/fxamacker/cbor/v2"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"

"github.com/onflow/flow-go/model/flow"
"github.com/onflow/flow-go/utils/rand"
"github.com/onflow/flow-go/utils/unittest"
)
@@ -15,7 +16,7 @@ import (
// out of range indices
func TestChunkList_ByIndex(t *testing.T) {
// creates a chunk list with the size of 10
var chunkList flow.ChunkList = make([]*flow.Chunk, 10)
var chunkList ChunkList = make([]*Chunk, 10)

// an out of index chunk by index
_, ok := chunkList.ByIndex(11)
@@ -36,14 +37,14 @@ func TestDistinctChunkIDs_EmptyChunks(t *testing.T) {
require.NotEqual(t, blockIdA, blockIdB)

// generates a chunk associated with each block id
chunkA := &flow.Chunk{
ChunkBody: flow.ChunkBody{
chunkA := &Chunk{
ChunkBody: ChunkBody{
BlockID: blockIdA,
},
}

chunkB := &flow.Chunk{
ChunkBody: flow.ChunkBody{
chunkB := &Chunk{
ChunkBody: ChunkBody{
BlockID: blockIdB,
},
}
@@ -83,7 +84,7 @@ func TestChunkList_Indices(t *testing.T) {
cl := unittest.ChunkListFixture(5, unittest.IdentifierFixture())
t.Run("empty chunk subset indices", func(t *testing.T) {
// subset of chunk list that is empty should return an empty list
subset := flow.ChunkList{}
subset := ChunkList{}
indices := subset.Indices()
require.Len(t, indices, 0)
})
@@ -100,7 +101,7 @@ func TestChunkList_Indices(t *testing.T) {
t.Run("multiple chunk subset indices", func(t *testing.T) {
// subset that only contains even chunk indices, should return
// a uint64 slice that only contains even chunk indices
subset := flow.ChunkList{cl[0], cl[2], cl[4]}
subset := ChunkList{cl[0], cl[2], cl[4]}
indices := subset.Indices()
require.Len(t, indices, 3)
require.Contains(t, indices, uint64(0), uint64(2), uint64(4))
@@ -111,7 +112,7 @@ func TestChunkIndexIsSet(t *testing.T) {

i, err := rand.Uint()
require.NoError(t, err)
chunk := flow.NewChunk(
chunk := NewChunk(
unittest.IdentifierFixture(),
int(i),
unittest.StateCommitmentFixture(),
@@ -129,7 +130,7 @@ func TestChunkNumberOfTxsIsSet(t *testing.T) {

i, err := rand.Uint32()
require.NoError(t, err)
chunk := flow.NewChunk(
chunk := NewChunk(
unittest.IdentifierFixture(),
3,
unittest.StateCommitmentFixture(),
@@ -146,7 +147,7 @@ func TestChunkTotalComputationUsedIsSet(t *testing.T) {

i, err := rand.Uint64()
require.NoError(t, err)
chunk := flow.NewChunk(
chunk := NewChunk(
unittest.IdentifierFixture(),
3,
unittest.StateCommitmentFixture(),
@@ -158,3 +159,135 @@ func TestChunkTotalComputationUsedIsSet(t *testing.T) {

assert.Equal(t, i, chunk.TotalComputationUsed)
}

// TODO doc
func TestChunkEncodeDecode(t *testing.T) {
chunk := unittest.ChunkFixture(unittest.IdentifierFixture(), 0)

t.Run("encode/decode preserves nil ServiceEventIndices", func(t *testing.T) {
chunk.ServiceEventIndices = nil
t.Run("json", func(t *testing.T) {
bz, err := json.Marshal(chunk)
require.NoError(t, err)
unmarshaled := new(Chunk)
err = json.Unmarshal(bz, unmarshaled)
require.NoError(t, err)
assert.Equal(t, chunk, unmarshaled)
assert.Nil(t, unmarshaled.ServiceEventIndices)
})
t.Run("cbor", func(t *testing.T) {
bz, err := cbor.Marshal(chunk)
require.NoError(t, err)
unmarshaled := new(Chunk)
err = cbor.Unmarshal(bz, unmarshaled)
require.NoError(t, err)
assert.Equal(t, chunk, unmarshaled)
assert.Nil(t, unmarshaled.ServiceEventIndices)
})
})
t.Run("encode/decode preserves empty but non-nil ServiceEventIndices", func(t *testing.T) {
chunk.ServiceEventIndices = []uint32{}
t.Run("json", func(t *testing.T) {
bz, err := json.Marshal(chunk)
require.NoError(t, err)
unmarshaled := new(Chunk)
err = json.Unmarshal(bz, unmarshaled)
require.NoError(t, err)
assert.Equal(t, chunk, unmarshaled)
assert.NotNil(t, unmarshaled.ServiceEventIndices)
})
t.Run("cbor", func(t *testing.T) {
bz, err := cbor.Marshal(chunk)
require.NoError(t, err)
unmarshaled := new(Chunk)
err = cbor.Unmarshal(bz, unmarshaled)
require.NoError(t, err)
assert.Equal(t, chunk, unmarshaled)
assert.NotNil(t, unmarshaled.ServiceEventIndices)
})
})
}

// TODO doc
func TestChunk_ModelVersions_EncodeDecode(t *testing.T) {
chunkFixture := unittest.ChunkFixture(unittest.IdentifierFixture(), 1)
chunkFixture.ServiceEventIndices = []uint32{1} // non-nil extra field

t.Run("writing v0 and reading v1 should yield nil for new field", func(t *testing.T) {
var chunkv0 chunkBodyV0
jordanschalm marked this conversation as resolved.
Show resolved Hide resolved
unittest.CopyStructure(t, chunkFixture.ChunkBody, chunkv0)

t.Run("json", func(t *testing.T) {
bz, err := json.Marshal(chunkv0)
require.NoError(t, err)

var unmarshaled ChunkBody
err = json.Unmarshal(bz, &unmarshaled)
require.NoError(t, err)
assert.Equal(t, chunkv0.EventCollection, unmarshaled.EventCollection)
assert.Equal(t, chunkv0.BlockID, unmarshaled.BlockID)
assert.Nil(t, unmarshaled.ServiceEventIndices)
})

t.Run("cbor", func(t *testing.T) {
bz, err := cbor.Marshal(chunkv0)
require.NoError(t, err)

var unmarshaled ChunkBody
err = cbor.Unmarshal(bz, &unmarshaled)
require.NoError(t, err)
assert.Equal(t, chunkv0.EventCollection, unmarshaled.EventCollection)
assert.Equal(t, chunkv0.BlockID, unmarshaled.BlockID)
assert.Nil(t, unmarshaled.ServiceEventIndices)
})
})
t.Run("writing v1 and reading v0 does not error", func(t *testing.T) {
chunkv1 := chunkFixture.ChunkBody
jordanschalm marked this conversation as resolved.
Show resolved Hide resolved
chunkv1.ServiceEventIndices = []uint32{0} // ensure non-nil ServiceEventIndices field

t.Run("json", func(t *testing.T) {
bz, err := json.Marshal(chunkv1)
require.NoError(t, err)

var unmarshaled chunkBodyV0
err = json.Unmarshal(bz, &unmarshaled)
require.NoError(t, err)
assert.Equal(t, chunkv1.EventCollection, unmarshaled.EventCollection)
assert.Equal(t, chunkv1.BlockID, unmarshaled.BlockID)
})
t.Run("cbor", func(t *testing.T) {
bz, err := cbor.Marshal(chunkv1)
require.NoError(t, err)

var unmarshaled chunkBodyV0
err = cbor.Unmarshal(bz, &unmarshaled)
require.NoError(t, err)
assert.Equal(t, chunkv1.EventCollection, unmarshaled.EventCollection)
assert.Equal(t, chunkv1.BlockID, unmarshaled.BlockID)
})
})
}

// TODO doc
func TestChunk_ModelVersions_IDConsistentAcrossVersions(t *testing.T) {
chunk := unittest.ChunkFixture(unittest.IdentifierFixture(), 1)
chunkBody := chunk.ChunkBody
var chunkv0 chunkBodyV0
unittest.CopyStructure(t, chunkBody, &chunkv0)

// A nil ServiceEventIndices fields indicates a prior model version.
// The ID calculation for the old and new model version should be the same.
t.Run("nil ServiceEventIndices fields", func(t *testing.T) {
chunkBody.ServiceEventIndices = nil
assert.Equal(t, chunkBody.BlockID, chunkv0.BlockID)
assert.Equal(t, MakeID(chunkv0), MakeID(chunkBody))
})
// A non-nil ServiceEventIndices fields indicates an up-to-date model version.
// The ID calculation for the old and new model version should be different,
// because the new model should include the ServiceEventIndices field value.
t.Run("non-nil ServiceEventIndices fields", func(t *testing.T) {
chunkBody.ServiceEventIndices = []uint32{}
assert.Equal(t, chunkBody.BlockID, chunkv0.BlockID)
assert.NotEqual(t, MakeID(chunkv0), MakeID(chunkBody))
})
}
16 changes: 16 additions & 0 deletions utils/unittest/encoding.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
package unittest

import (
"testing"

"github.com/fxamacker/cbor/v2"
"github.com/stretchr/testify/require"
)

// TODO doc
func CopyStructure(t *testing.T, src, dst any) {
bz, err := cbor.Marshal(src)
require.NoError(t, err)
err = cbor.Unmarshal(bz, dst)
require.NoError(t, err)
}

Unchanged files with check annotations Beta

package flow

Check failure on line 1 in model/flow/account.go

GitHub Actions / Lint (./)

: import cycle not allowed in test (typecheck)
import (
"encoding/json"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/onflow/flow-go/model/flow"

Check failure on line 13 in model/flow/account_encoder_test.go

GitHub Actions / Lint (./)

could not import github.com/onflow/flow-go/model/flow (-: import cycle not allowed in test) (typecheck)
)
type legacyAccountPublicKeyWrapper struct {
t.Run("json", func(t *testing.T) {
t.Run("specific event types", func(t *testing.T) {
// EpochSetup
assertJsonConvert(t, setup, comparePubKey)

Check failure on line 41 in model/flow/service_event_test.go

GitHub Actions / Lint (./)

cannot infer T (model/flow/service_event_test.go:189:24) (typecheck)
// EpochCommit
assertJsonConvert(t, commit, comparePubKey)

Check failure on line 44 in model/flow/service_event_test.go

GitHub Actions / Lint (./)

cannot infer T (model/flow/service_event_test.go:189:24) (typecheck)
// EpochRecover
assertJsonConvert(t, epochRecover, comparePubKey)

Check failure on line 47 in model/flow/service_event_test.go

GitHub Actions / Lint (./)

cannot infer T (model/flow/service_event_test.go:189:24) (typecheck)
// VersionBeacon
assertJsonConvert(t, versionBeacon)
t.Run("generic type", func(t *testing.T) {
// EpochSetup
assertJsonGenericConvert(t, setup, comparePubKey)

Check failure on line 64 in model/flow/service_event_test.go

GitHub Actions / Lint (./)

cannot infer T (model/flow/service_event_test.go:200:31) (typecheck)
// EpochCommit
assertJsonGenericConvert(t, commit, comparePubKey)

Check failure on line 67 in model/flow/service_event_test.go

GitHub Actions / Lint (./)

cannot infer T (model/flow/service_event_test.go:200:31) (typecheck)
// EpochRecover
assertJsonGenericConvert(t, epochRecover, comparePubKey)

Check failure on line 70 in model/flow/service_event_test.go

GitHub Actions / Lint (./)

cannot infer T (model/flow/service_event_test.go:200:31) (typecheck)
// VersionBeacon
assertJsonGenericConvert(t, versionBeacon)
t.Run("msgpack", func(t *testing.T) {
t.Run("specific event types", func(t *testing.T) {
// EpochSetup
assertMsgPackConvert(t, setup, comparePubKey)

Check failure on line 89 in model/flow/service_event_test.go

GitHub Actions / Lint (./)

cannot infer T (model/flow/service_event_test.go:213:27) (typecheck)
// EpochCommit
assertMsgPackConvert(t, commit, comparePubKey)

Check failure on line 92 in model/flow/service_event_test.go

GitHub Actions / Lint (./)

cannot infer T (model/flow/service_event_test.go:213:27) (typecheck)
// EpochRecover
assertMsgPackConvert(t, epochRecover, comparePubKey)