Skip to content

Commit

Permalink
uses permalink and link id for recurring references
Browse files Browse the repository at this point in the history
  • Loading branch information
staheri14 committed Mar 7, 2023
1 parent 93a70ae commit 22e737c
Showing 1 changed file with 27 additions and 15 deletions.
42 changes: 27 additions & 15 deletions docs/spec/nmt-wrapper.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Namespaced Merkle Tree Wrapper
## Abstract
In Celestia, block transactions are grouped into identical-size shares and arranged in a `k` by `k` matrix, called [original data square](https://github.com/celestiaorg/celestia-app/blob/specs-staging/specs/src/specs/data_structures.md#arranging-available-data-into-shares).
In Celestia, block transactions are grouped into identical-size shares and arranged in a `k` by `k` matrix, called [original data square](originalds-link).
The rows and columns of the original data square are then extended using a 2D Reed-Solomon coding scheme.
The [extended version](https://github.com/celestiaorg/celestia-app/blob/specs-staging/specs/src/specs/data_structures.md#2d-reed-solomon-encoding-scheme) is called extended data square, that is a `2k` by `2k` square consisting of `4` quadrants namely, `Q0`, `Q1`, `Q2`, and `Q3`.
The [extended version](reedsolomon-link) is called extended data square, that is a `2k` by `2k` square consisting of `4` quadrants namely, `Q0`, `Q1`, `Q2`, and `Q3`.
Figure 1 provides an illustration of a sample data square and its extended version.
`Q0` corresponds to the original data square.
`Q1` and `Q2` represent the horizontal and vertical extensions of `Q0`, respectively.
Expand All @@ -21,25 +21,25 @@ Figure 1 provides an illustration of a sample data square and its extended versi
```
Figure 1.

Each row and column of the extended data square is modeled by a [Namespace Merkle Tree](https://github.com/celestiaorg/nmt/blob/master/spec/nmt.md).
Each row and column of the extended data square is modeled by a [Namespace Merkle Tree](nmtlink).
NMTs require the data items they represent to be namespaced, which means that the shares within each row or column of the extended data square must be namespaced before being added to the NMT.
This is where the [Namespaced MerkleTree Wrapper](https://github.com/celestiaorg/celestia-app/blob/main/pkg/wrapper/nmt_wrapper.go) comes into play.
It is a data structure that wraps around the [Namespaced Merkle Tree](https://github.com/celestiaorg/nmt/blob/master/spec/nmt.md) and ensures the proper namespaces are prepended to the shares before they are added to their respective row or column NMT.
This is where the Namespaced Merkle Tree Wrapper, more specifically, [Erasured Namespaced Merkle Tree](nmtwrapper-link) comes into play.
It is a data structure that wraps around the [Namespaced Merkle Tree](nmtlink) and ensures the proper namespaces are prepended to the shares before they are added to their respective row or column NMT.
In this specification, we elaborate on the design and structure of the Namespace Merkle Tree wrapper.



## Namespaced Merkle Tree Wrapper
Namespaced Merkle Tree Wrapper is used to represent a row or column of Celestia extended data square.
At its core, it is an [NMT](https://github.com/celestiaorg/nmt/blob/master/spec/nmt.md) and is defined by a similar set of parameters.
Namely, the [namespace ID size `NamespaceIDSize`](https://github.com/celestiaorg/nmt/blob/master/spec/nmt.md#nmt-data-structure),
the underlying hash function for the digest calculation of the [namespaced hashes](https://github.com/celestiaorg/nmt/blob/master/spec/nmt.md#namespaced-hash),
and the [`IgnoreMaxNamespace` flag](https://github.com/celestiaorg/nmt/blob/master/spec/nmt.md#ignore-max-namespace) which dictates how namespace range of non-leaf nodes are calculated.
In addition, the NMT wrapper is configured with the data square size `SqaureSize` (`k` in the above example), and the index of the row or column it represents `AxisIndex` which is a value in `[0, SquareSize)`.
At its core, it is an [NMT](nmtlink) and is defined by a similar set of parameters.
Namely, the [namespace ID size `NamespaceIDSize`](nmt-ds-link),
the underlying hash function for the digest calculation of the [namespaced hashes](nmt-hash-link),
and the [`IgnoreMaxNamespace` flag](nmt-ignoremax-link) which dictates how namespace range of non-leaf nodes are calculated.
In addition, the NMT wrapper is configured with the original data square size `SqaureSize` (`k` in the above example), and the index of the row or column it represents `AxisIndex` which is a value in `[0, 2*SquareSize)`.
These additional configurations are used to determine the namespace ID of the shares that the NMT wrapper represents based on the quadrants to which they belong.

NMT wrapper supports [Merkle inclusion proof](#link-to-the-nmt-spec-for-the-inclusion-proof) for the given share index and [Merkle range proof](#link-to-the-nmt-spec-for-the-range-proof) for a range of share indices.
It extends the NMT data insertion behaviour (i.e., the [`Push` method](https://github.com/celestiaorg/nmt/blob/master/spec/nmt.md#add-leaves)) to prepend shares with proper namespace before inclusion in the tree.
NMT wrapper supports [Merkle inclusion proof](#nmt-inclusion-proof) for the given share index and [Merkle range proof](#nmt-range-proof) for a range of share indices.
It extends the NMT data insertion behaviour (i.e., the [`Push` method](nmt-add-leaves-link)) to prepend shares with proper namespace before inclusion in the tree.

### Namespace ID assignment to the shares of an extended data square
To understand the NMT wrapper data structure, it's important to first understand how shares are namespaced in Celestia.
Expand All @@ -51,8 +51,8 @@ Thus, the namespace ID of a share in `Q0` can be extracted from the first `Names
However, shares in `Q1`, `Q2`, and `Q3` (that are erasure-coded versions of `Q0` and known as parity shares) do not have any namespace IDs by default.
These shares must be assigned a reserved namespace ID, which is called `ParitySharesNamespaceID`.
The `ParitySharesNamespaceID` corresponds to the lexicographically last namespace ID that can be represented by `NamespaceIDSize` bytes.
"If `NamespaceIDSize` is `8`, then the value of `ParitySharesNamespaceID` is `2^64-1`, which is equivalent to `0xFFFFFFFFFFFFFFFF`.
In Celestia, the values for `NamespaceIDSize` and `ParitySharesNamespaceID` can be found in [`NAMESPACE_ID_BYTES`](https://github.com/celestiaorg/celestia-app/blob/specs-staging/specs/src/specs/consensus.md#constants) constant and the [`PARITY_SHARE_NAMESPACE_ID`](https://github.com/celestiaorg/celestia-app/blob/specs-staging/specs/src/specs/consensus.md#reserved-namespace-ids), respectively.
If `NamespaceIDSize` is `8`, then the value of `ParitySharesNamespaceID` is `2^64-1`, which is equivalent to `0xFFFFFFFFFFFFFFFF`.
In Celestia, the values for `NamespaceIDSize` and `ParitySharesNamespaceID` can be found in [`NAMESPACE_ID_BYTES`](celestia-constants-link) constant and the [`PARITY_SHARE_NAMESPACE_ID`](celestia-consensus-link), respectively.

### NMT Wrapper Data Insertion
The NMT wrapper insertion logic is governed by the same rules as the NMT insertion logic.
Expand Down Expand Up @@ -80,4 +80,16 @@ One namespace ID is located in the first `NamespaceIDSize` bytes, while the othe
- Namespaced Merkle tree specifications: https://github.com/celestiaorg/nmt/blob/master/spec/nmt.md
- Celestia original data square specification: https://github.com/celestiaorg/celestia-app/blob/specs-staging/specs/src/specs/data_structures.md#arranging-available-data-into-shares
- Celestia constants: https://github.com/celestiaorg/celestia-app/blob/specs-staging/specs/src/specs/consensus.md#constants
- Celestia reserved namespace IDs: https://github.com/celestiaorg/celestia-app/blob/specs-staging/specs/src/specs/consensus.md#reserved-namespace-ids
- Celestia reserved namespace IDs: https://github.com/celestiaorg/celestia-app/blob/specs-staging/specs/src/specs/consensus.md#reserved-namespace-ids


[nmtlink]: https://github.com/celestiaorg/nmt/blob/master/spec/nmt.md
[nmtwrapper-link]: https://github.com/celestiaorg/celestia-app/blob/main/pkg/wrapper/nmt_wrapper.go
[nmt-ds-link]: https://github.com/celestiaorg/nmt/blob/master/spec/nmt.md#nmt-data-structure
[nmt-hash-link]: https://github.com/celestiaorg/nmt/blob/master/spec/nmt.md#namespaced-hash
[nmt-ignoremax-link]: https://github.com/celestiaorg/nmt/blob/master/spec/nmt.md#ignore-max-namespace
[nmt-add-leaves-link]: https://github.com/celestiaorg/nmt/blob/master/spec/nmt.md#add-leaves
[celestia-constants-link]: https://github.com/celestiaorg/celestia-app/blob/c09843d07d4c3842753138de96b304b4866e8f5d/specs/src/specs/consensus.md#constants
[celestia-consensus-link]: https://github.com/celestiaorg/celestia-app/blob/c09843d07d4c3842753138de96b304b4866e8f5d/specs/src/specs/consensus.md#reserved-namespace-ids
[reedsolomon-link]: https://github.com/celestiaorg/celestia-app/blob/c09843d07d4c3842753138de96b304b4866e8f5d/specs/src/specs/data_structures.md#2d-reed-solomon-encoding-scheme
[originalds-link]: https://github.com/celestiaorg/celestia-app/blob/c09843d07d4c3842753138de96b304b4866e8f5d/specs/src/specs/data_structures.md?plain=1#L494

0 comments on commit 22e737c

Please sign in to comment.