Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat!: decrease namespace from 33 to 29 bytes #1771

Merged
merged 8 commits into from
May 18, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -201,5 +201,5 @@ require (
replace (
github.com/cosmos/cosmos-sdk => github.com/celestiaorg/cosmos-sdk v1.13.0-sdk-v0.46.11
github.com/gogo/protobuf => github.com/regen-network/protobuf v1.3.3-alpha.regen.1
github.com/tendermint/tendermint => github.com/celestiaorg/celestia-core v1.19.0-tm-v0.34.27
github.com/tendermint/tendermint => github.com/celestiaorg/celestia-core v1.20.0-tm-v0.34.27
)
4 changes: 2 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -169,8 +169,8 @@ github.com/btcsuite/winsvc v1.0.0/go.mod h1:jsenWakMcC0zFBFurPLEAyrnc/teJEM1O46f
github.com/bwesterb/go-ristretto v1.2.0/go.mod h1:fUIoIZaG73pV5biE2Blr2xEzDoMj7NFEuV9ekS419A0=
github.com/c-bata/go-prompt v0.2.2/go.mod h1:VzqtzE2ksDBcdln8G7mk2RX9QyGjH+OVqOCSiVIqS34=
github.com/casbin/casbin/v2 v2.1.2/go.mod h1:YcPU1XXisHhLzuxH9coDNf2FbKpjGlbCg3n9yuLkIJQ=
github.com/celestiaorg/celestia-core v1.19.0-tm-v0.34.27 h1:GLdDJRu1fRSMft4IqQz4/x/H1U3eN2TFlYbAycbSiN4=
github.com/celestiaorg/celestia-core v1.19.0-tm-v0.34.27/go.mod h1:8PbX2OIPehldawXWAzNWPxBPnfFtcYtjHecE45b2Beg=
github.com/celestiaorg/celestia-core v1.20.0-tm-v0.34.27 h1:zyAhkcRNsYvt1FglnzOyw3WV9fOn4aXaXUsz87NAunk=
github.com/celestiaorg/celestia-core v1.20.0-tm-v0.34.27/go.mod h1:8PbX2OIPehldawXWAzNWPxBPnfFtcYtjHecE45b2Beg=
github.com/celestiaorg/cosmos-sdk v1.13.0-sdk-v0.46.11 h1:Rd5EvJx1nG3KurBspVN51RVmvif0Lp2UVURbG2ad3Cs=
github.com/celestiaorg/cosmos-sdk v1.13.0-sdk-v0.46.11/go.mod h1:xCG6OUkJy5KUMEg20Zk010lra9XjkmKS3+bk0wp7bd8=
github.com/celestiaorg/merkletree v0.0.0-20210714075610-a84dc3ddbbe4 h1:CJdIpo8n5MFP2MwK0gSRcOVlDlFdQJO1p+FqdxYzmvc=
Expand Down
2 changes: 1 addition & 1 deletion pkg/appconsts/appconsts.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ const (
NamespaceVersionSize = 1

// NamespaceIDSize is the size of a namespace ID in bytes.
NamespaceIDSize = 32
NamespaceIDSize = 28
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[non blocking]
This const is also defined in core, wouldn't it make sense to use it here instead of rehardcoding it here?


// NamespaceSize is the size of a namespace (version + ID) in bytes.
NamespaceSize = NamespaceVersionSize + NamespaceIDSize
Expand Down
6 changes: 3 additions & 3 deletions pkg/da/data_availability_header_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ func TestNilDataAvailabilityHeaderHashDoesntCrash(t *testing.T) {

func TestMinDataAvailabilityHeader(t *testing.T) {
dah := MinDataAvailabilityHeader()
expectedHash := []byte{0xe9, 0x5, 0x28, 0x49, 0xf, 0x1d, 0x51, 0x67, 0x29, 0x2c, 0x1f, 0x1b, 0x83, 0xe1, 0x74, 0x2a, 0x27, 0x48, 0x17, 0x34, 0x12, 0xc9, 0x1d, 0xf7, 0xdd, 0x1, 0x96, 0x78, 0xa4, 0x62, 0xb9, 0x77}
expectedHash := []byte{0x3d, 0x96, 0xb7, 0xd2, 0x38, 0xe7, 0xe0, 0x45, 0x6f, 0x6a, 0xf8, 0xe7, 0xcd, 0xf0, 0xa6, 0x7b, 0xd6, 0xcf, 0x9c, 0x20, 0x89, 0xec, 0xb5, 0x59, 0xc6, 0x59, 0xdc, 0xaa, 0x1f, 0x88, 0x3, 0x53}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Non Blocking][question]
I am wondering how these expected hashes are generated?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good follow up to document this imo.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am wondering how these expected hashes are generated?

I generate them by running the test after a share encoding change and copying the actual output to the expected output. Related doc:

// TestCreateCommitment will fail if a change is made to share encoding or how
// the commitment is calculated. If this is the case, the expected commitment
// bytes will need to be updated.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool 👍

require.Equal(t, expectedHash, dah.hash)
require.NoError(t, dah.ValidateBasic())
}
Expand All @@ -42,13 +42,13 @@ func TestNewDataAvailabilityHeader(t *testing.T) {
tests := []test{
{
name: "typical",
expectedHash: []byte{0x5b, 0x27, 0x3e, 0x3a, 0x5d, 0x9e, 0x90, 0x25, 0x58, 0x21, 0xb7, 0xe0, 0x4d, 0x4b, 0xaa, 0xde, 0x37, 0xa6, 0x6f, 0xcc, 0xd, 0x16, 0x6f, 0x9e, 0xe0, 0x7f, 0xbe, 0x8, 0xb4, 0x41, 0xc8, 0xa6},
expectedHash: []byte{0xb5, 0x6e, 0x4d, 0x25, 0x1a, 0xc2, 0x66, 0xf4, 0xb9, 0x1c, 0xc5, 0x46, 0x4b, 0x3f, 0xc7, 0xef, 0xcb, 0xdc, 0x88, 0x80, 0x64, 0x64, 0x74, 0x96, 0xd1, 0x31, 0x33, 0xf0, 0xdc, 0x65, 0xac, 0x25},
squareSize: 2,
shares: generateShares(4),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] Can we use 2*2 instead of 4 in the generateShares(4), it is more clear how 4 is calculated and why.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure! #1788

},
{
name: "max square size",
expectedHash: []byte{0xce, 0x5c, 0xf3, 0xc9, 0x15, 0xeb, 0xbf, 0xb0, 0x67, 0xe1, 0xa5, 0x97, 0x35, 0xf3, 0x25, 0x7b, 0x1c, 0x47, 0x74, 0x1f, 0xec, 0x6a, 0x33, 0x19, 0x7f, 0x8f, 0xc2, 0x4a, 0xe, 0xe2, 0xbe, 0x73},
expectedHash: []byte{0xb, 0xd3, 0xab, 0xee, 0xac, 0xfb, 0xb0, 0xb9, 0x2d, 0xfb, 0xda, 0xc4, 0xa1, 0x54, 0x86, 0x8e, 0x3c, 0x4e, 0x79, 0x66, 0x6f, 0x7f, 0xcf, 0x6c, 0x62, 0xb, 0xb9, 0xd, 0xd3, 0xa0, 0xdc, 0xf0},
squareSize: appconsts.MaxSquareSize,
shares: generateShares(appconsts.MaxSquareSize * appconsts.MaxSquareSize),
},
Expand Down
2 changes: 1 addition & 1 deletion pkg/namespace/consts.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ const (

// NamespaceZeroPrefixSize is the number of `0` bytes that are prefixed to
// namespace IDs for version 0.
NamespaceVersionZeroPrefixSize = 22
NamespaceVersionZeroPrefixSize = 18

// NamespaceVersionZeroIDSize is the number of bytes available for
// user-specified namespace ID in a namespace ID for version 0.
Expand Down
8 changes: 4 additions & 4 deletions pkg/shares/doc.go
Original file line number Diff line number Diff line change
Expand Up @@ -33,12 +33,12 @@
// For the first share of a sequence:
//
// | namespace_version | namespace_id | info_byte | sequence_length | sequence_data |
// | 1 byte | 32 bytes | 1 byte | 4 bytes | remaining bytes of share |
// | 1 byte | 28 bytes | 1 byte | 4 bytes | remaining bytes of share |
//
// For continuation share of a sequence:
//
// | namespace_version | namespace_id | info_byte | sequence_data |
// | 1 byte | 32 bytes | 1 byte | remaining bytes of share |
// | 1 byte | 28 bytes | 1 byte | remaining bytes of share |
//
// The remaining bytes depend on the share type.
//
Expand All @@ -51,12 +51,12 @@
// For the first compact share:
//
// | namespace_version | namespace_id | info_byte | sequence_length | location_of_first_unit | transactions or intermediate state roots |
// | 1 byte | 32 bytes | 1 byte | 4 bytes | 4 bytes | remaining bytes of share |
// | 1 byte | 28 bytes | 1 byte | 4 bytes | 4 bytes | remaining bytes of share |
//
// For continuation compact share:
//
// | namespace_version | namespace_id | info_byte | location_of_first_unit | transactions or intermediate state roots |
// | 1 byte | 32 bytes | 1 byte | 4 bytes | remaining bytes of share |
// | 1 byte | 28 bytes | 1 byte | 4 bytes | remaining bytes of share |
//
// Notes
// - All shares in a reserved namespace belong to one sequence.
Expand Down
2 changes: 1 addition & 1 deletion pkg/shares/share_sequence_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ func Test_sparseSharesNeeded(t *testing.T) {
{appconsts.FirstSparseShareContentSize + appconsts.ContinuationCompactShareContentSize*99, 100},
{1000, 3},
{10000, 21},
{100000, 210},
{100000, 208},
rootulp marked this conversation as resolved.
Show resolved Hide resolved
}
for _, tc := range testCases {
got := SparseSharesNeeded(tc.sequenceLen)
Expand Down
26 changes: 13 additions & 13 deletions pkg/shares/share_splitting_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ func TestSplitTxs_forTxShares(t *testing.T) {
[]byte{
0x1, // info byte
0x0, 0x0, 0x0, 0x2, // 1 byte (unit) + 1 byte (unit length) = 2 bytes sequence length
0x0, 0x0, 0x0, 0x2a, // reserved bytes
0x0, 0x0, 0x0, 0x26, // reserved bytes
rootulp marked this conversation as resolved.
Show resolved Hide resolved
0x1, // unit length of first transaction
0xa, // data of first transaction
}...,
Expand All @@ -57,7 +57,7 @@ func TestSplitTxs_forTxShares(t *testing.T) {
[]byte{
0x1, // info byte
0x0, 0x0, 0x0, 0x4, // 2 bytes (first transaction) + 2 bytes (second transaction) = 4 bytes sequence length
0x0, 0x0, 0x0, 0x2a, // reserved bytes
0x0, 0x0, 0x0, 0x26, // reserved bytes
0x1, // unit length of first transaction
0xa, // data of first transaction
0x1, // unit length of second transaction
Expand All @@ -78,7 +78,7 @@ func TestSplitTxs_forTxShares(t *testing.T) {
[]byte{
0x1, // info byte
0x0, 0x0, 0x2, 0x2, // 512 (unit) + 2 (unit length) = 514 sequence length
0x0, 0x0, 0x0, 0x2a, // reserved bytes
0x0, 0x0, 0x0, 0x26, // reserved bytes
128, 4, // unit length of transaction is 512
}...,
),
Expand All @@ -94,7 +94,7 @@ func TestSplitTxs_forTxShares(t *testing.T) {
0x0, 0x0, 0x0, 0x0, // reserved bytes
}...,
),
bytes.Repeat([]byte{0xc}, 44)..., // continuation data of transaction
bytes.Repeat([]byte{0xc}, 40)..., // continuation data of transaction
),
},
),
Expand All @@ -110,7 +110,7 @@ func TestSplitTxs_forTxShares(t *testing.T) {
[]byte{
0x1, // info byte
0x0, 0x0, 0x2, 0x4, // 2 bytes (first transaction) + 514 bytes (second transaction) = 516 bytes sequence length
0x0, 0x0, 0x0, 0x2a, // reserved bytes
0x0, 0x0, 0x0, 0x26, // reserved bytes
1, // unit length of first transaction
0xa, // data of first transaction
128, 4, // unit length of second transaction is 512
Expand All @@ -128,7 +128,7 @@ func TestSplitTxs_forTxShares(t *testing.T) {
0x0, 0x0, 0x0, 0x0, // reserved bytes
}...,
),
bytes.Repeat([]byte{0xc}, 46)..., // continuation data of second transaction
bytes.Repeat([]byte{0xc}, 42)..., // continuation data of second transaction
),
},
),
Expand All @@ -144,7 +144,7 @@ func TestSplitTxs_forTxShares(t *testing.T) {
[]byte{
0x1, // info byte
0x0, 0x0, 0x2, 0x4, // 514 bytes (first transaction) + 2 bytes (second transaction) = 516 bytes sequence length
0x0, 0x0, 0x0, 0x2a, // reserved bytes
0x0, 0x0, 0x0, 0x26, // reserved bytes
128, 4, // unit length of first transaction is 512
}...,
),
Expand All @@ -156,10 +156,10 @@ func TestSplitTxs_forTxShares(t *testing.T) {
appns.TxNamespace.Bytes(),
[]byte{
0x0, // info byte
0x0, 0x0, 0x0, 0x52, // reserved bytes
0x0, 0x0, 0x0, 0x4a, // reserved bytes
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion, non-blocking] Could we include visual representations illustrating how the data is organized within their respective shares? by visual representation I mean something similar to the ones in this example.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0xc, 0xc, 0xc, 0xc, 0xc, 0xc, 0xc, 0xc, 0xc, 0xc, 0xc, 0xc, 0xc, 0xc, 0xc, 0xc, // continuation data of first transaction
0xc, 0xc, 0xc, 0xc, 0xc, 0xc, 0xc, 0xc, 0xc, 0xc, 0xc, 0xc, 0xc, 0xc, 0xc, 0xc, // continuation data of first transaction
0xc, 0xc, 0xc, 0xc, 0xc, 0xc, 0xc, 0xc, 0xc, 0xc, 0xc, 0xc, // continuation data of first transaction
0xc, 0xc, 0xc, 0xc, 0xc, 0xc, 0xc, 0xc, // continuation data of first transaction
1, // unit length of second transaction
0xa, // data of second transaction
}...,
Expand Down Expand Up @@ -196,7 +196,7 @@ func TestSplitTxs(t *testing.T) {
[]byte{
0x1, // info byte
0x0, 0x0, 0x0, 0x2, // 1 byte (unit) + 1 byte (unit length) = 2 bytes sequence length
0x0, 0x0, 0x0, 0x2a, // reserved bytes
0x0, 0x0, 0x0, 0x26, // reserved bytes
0x1, // unit length of first transaction
0xa, // data of first transaction
}...,
Expand All @@ -214,7 +214,7 @@ func TestSplitTxs(t *testing.T) {
[]uint8{
0x1, // info byte
0x0, 0x0, 0x0, 13, // 1 byte (unit) + 1 byte (unit length) = 2 bytes sequence length
0x0, 0x0, 0x0, 0x2a, // reserved bytes
0x0, 0x0, 0x0, 0x26, // reserved bytes
12, // unit length of first transaction
0xa, 0x1, 0xb, 0x12, 0x1, 0xa, 0x1a, 0x4, 0x49, 0x4e, 0x44, 0x58, // data of first transaction
}...,
Expand All @@ -230,7 +230,7 @@ func TestSplitTxs(t *testing.T) {
[]uint8{
0x1, // info byte
0x0, 0x0, 0x2, 0x2, // 512 (unit) + 2 (unit length) = 514 sequence length
0x0, 0x0, 0x0, 0x2a, // reserved bytes
0x0, 0x0, 0x0, 0x26, // reserved bytes
128, 4, // unit length of transaction is 512
}...,
),
Expand All @@ -245,7 +245,7 @@ func TestSplitTxs(t *testing.T) {
0x0, 0x0, 0x0, 0x0, // reserved bytes
}...,
),
bytes.Repeat([]byte{0xc}, 44)..., // continuation data of transaction
bytes.Repeat([]byte{0xc}, 40)..., // continuation data of transaction
),
},
),
Expand Down
4 changes: 2 additions & 2 deletions pkg/shares/split_compact_shares_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ func TestExport_write(t *testing.T) {
[]byte{
0x1, // info byte
0x0, 0x0, 0x0, 0x1, // sequence len
0x0, 0x0, 0x0, 0x2a, // reserved bytes
0x0, 0x0, 0x0, 0x26, // reserved bytes
0xf, // data
}...,
),
Expand All @@ -79,7 +79,7 @@ func TestExport_write(t *testing.T) {
[]byte{
0x1, // info byte
0x0, 0x0, 0x2, 0x0, // sequence len
0x0, 0x0, 0x0, 0x2a, // reserved bytes
0x0, 0x0, 0x0, 0x26, // reserved bytes
}...,
)}, 0xf)

Expand Down
22 changes: 11 additions & 11 deletions specs/src/specs/consensus.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,28 +25,28 @@
| `MAX_GRAFFITI_BYTES` | `uint64` | `32` | `byte` | Maximum size of transaction graffiti, in bytes. |
| `MAX_VALIDATORS` | `uint16` | `64` | | Maximum number of active validators. |
| `NAMESPACE_VERSION_SIZE` | `int` | `1` | `byte` | Size of namespace version in bytes. |
| `NAMESPACE_ID_SIZE` | `int` | `32` | `byte` | Size of namespace ID in bytes. |
| `NAMESPACE_SIZE` | `int` | `33` | `byte` | Size of namespace in bytes. |
| `NAMESPACE_ID_SIZE` | `int` | `28` | `byte` | Size of namespace ID in bytes. |
| `NAMESPACE_SIZE` | `int` | `29` | `byte` | Size of namespace in bytes. |
| `NAMESPACE_ID_MAX_RESERVED` | `uint64` | `255` | | Value of maximum reserved namespace (inclusive). 1 byte worth of IDs. |
| `SEQUENCE_BYTES` | `uint64` | `4` | `byte` | The number of bytes used to store the sequence length in the first share of a sequence |
| `SHARE_INFO_BYTES` | `uint64` | `1` | `byte` | The number of bytes used for [share](data_structures.md#share) information |
| `SHARE_RESERVED_BYTES` | `uint64` | `4` | `byte` | The number of bytes used to store the location of the first unit in a compact share. Must be able to represent any integer up to and including `SHARE_SIZE - 1`. |
| `SHARE_SIZE` | `uint64` | `512` | `byte` | Size of transaction and blob [shares](data_structures.md#share), in bytes. |
| `SHARE_SIZE` | `uint64` | `512` | `byte` | Size of transaction and blob [shares](data_structures.md#share), in bytes. |
| `STATE_SUBTREE_RESERVED_BYTES` | `uint64` | `1` | `byte` | Number of bytes reserved to identify state subtrees. |
| `UNBONDING_DURATION` | `uint32` | | `block` | Duration, in blocks, for unbonding a validator or delegation. |
| `VERSION_APP` | `uint64` | `1` | | Version of the Celestia application. Breaking changes (hard forks) must update this parameter. |
| `VERSION_BLOCK` | `uint64` | `1` | | Version of the Celestia chain. Breaking changes (hard forks) must update this parameter. |

### Reserved Namespaces

| name | type | value | description |
|-------------------------------------|-------------|------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------|
| `TRANSACTION_NAMESPACE` | `Namespace` | `0x000000000000000000000000000000000000000000000000000000000000000001` | Transactions: requests that modify the state. |
| `INTERMEDIATE_STATE_ROOT_NAMESPACE` | `Namespace` | `0x000000000000000000000000000000000000000000000000000000000000000002` | Intermediate state roots, committed after every transaction. |
| `RESERVED_PADDING_NAMESPACE` | `Namespace` | `0x0000000000000000000000000000000000000000000000000000000000000000FF` | Padding after all reserved namespaces but before blobs. |
| `MAX_RESERVED_NAMESPACE` | `Namespace` | `0x0000000000000000000000000000000000000000000000000000000000000000FF` | Max reserved namespace is lexicographically the largest namespace that is reserved for protocol use. |
| `TAIL_PADDING_NAMESPACE` | `Namespace` | `0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFE` | Tail padding for blobs: padding after all blobs to fill up the original data square. |
| `PARITY_SHARE_NAMESPACE` | `Namespace` | `0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF` | Parity shares: extended shares in the available data matrix. |
| name | type | value | description |
|-------------------------------------|-------------|----------------------------------------------------------------|------------------------------------------------------------------------------------------------------|
| `TRANSACTION_NAMESPACE` | `Namespace` | `0x0000000000000000000000000000000000000000000000000000000001` | Transactions: requests that modify the state. |
| `INTERMEDIATE_STATE_ROOT_NAMESPACE` | `Namespace` | `0x0000000000000000000000000000000000000000000000000000000002` | Intermediate state roots, committed after every transaction. |
| `RESERVED_PADDING_NAMESPACE` | `Namespace` | `0x00000000000000000000000000000000000000000000000000000000FF` | Padding after all reserved namespaces but before blobs. |
| `MAX_RESERVED_NAMESPACE` | `Namespace` | `0x00000000000000000000000000000000000000000000000000000000FF` | Max reserved namespace is lexicographically the largest namespace that is reserved for protocol use. |
| `TAIL_PADDING_NAMESPACE` | `Namespace` | `0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFE` | Tail padding for blobs: padding after all blobs to fill up the original data square. |
| `PARITY_SHARE_NAMESPACE` | `Namespace` | `0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF` | Parity shares: extended shares in the available data matrix. |

### Rewards and Penalties

Expand Down
Loading