DAS phase 1 #2146

vbuterin · 2020-12-09T07:30:19Z

Puts data availability sampling based on the ideas here into a pull request.

…fining the new SSZ class with the new field classes

specs/phase1/beacon-chain.md

Co-authored-by: dankrad <dankrad@ethereum.org>

specs/phase1/beacon-chain.md

Co-authored-by: dankrad <dankrad@ethereum.org>

dankrad · 2020-12-14T22:35:42Z

specs/phase1/beacon-chain.md

+| Name | Value | Unit | Description |
+| - | - | - | - |
+| `MAX_GASPRICE` | `Gwei(2**24)` (= 16,777,216) | Gwei | Max gasprice charged for an TARGET-sized shard block |  
+| `MIN_GASPRICE` | `Gwei(2**3)` (= 8) | Gwei | Min gasprice charged for an TARGET-sized shard block |


Denominating in GWei seems coarse. Given that it's a 64 bit integer, it would be easy to increase resolution to Wei. Have you checked the dynamics of this when the gas price is at the lower end of the scale? Gas might become unresponsive/take longer to adjust

It's not that coarse! Because it's not gwei per gas, it's gwei for the entire contents of the block. So the lowest possible gasprice is to literally pay (at current prices) $0.00000476 to fill an entire block.

I'm just saying that at the lower limit it may behave erratically. Maybe we want a higher lower limit then?

I just made a patch so that the gasprice delta is minimum 1, so it won't get stuck even at lower levels. Would that work? This is how the EIP 1559 proposal works on the eth1 side, and it seems a good idea to use similar formulas for both.

I agree that this should pretty safely fix the issue at the low end of the spectrum. Question about the upper limit though -- that seems fairly low, only 0.017 ETH per block. That could very plausibly be breached. Current Eth1 blocks have already brought in several ETH per block.

(I guess this would mean that we have to check that a proposer has enough balance before creating a block)

My current suggestion is to allow up to ~8 ETH gas (meaning ~16 ETH at full size) and block validators with insufficient balance from proposer selection.

specs/phase1/beacon-chain.md

dankrad · 2020-12-14T22:44:20Z

specs/phase1/beacon-chain.md

+                c for c in state.previous_epoch_pending_shard_headers if
+                (c.slot, c.shard, c.confirmed) == (slot, shard, True)
+            ]
+            if confirmed_candidates:


Why should the proposer only be charged for confirmed blocks? If they are doing their job reasonably, then they should be able to get their proposal confirmed, so it would be fair to always charge them. After all, an unavailable block might cause as much, if not more work on attesters.

Good question! Need to think through the consequences of this. The status quo in ethereum is certainly "you pay only if you get included", though I can see the arguments for changing it.

I think the main consequence is that we introduce a griefing vector by a committee, to conspire to not confirm a shard block in order to charge a proposer the gas fee without a reward.

Second possible problem could be that under condition where you consider it less likely that a block will be confirmed, e.g. when latency is high, it would discourage block production. In fact this threshold could be quite low: if we expect that a block producer makes 10% profit, then any confirmation probability under 90% would discourage block production.

Charging for unconfirmed blocks would make equivocation to fill up proposals more expensive (#2146 (comment))

specs/phase1/beacon-chain.md

…it what's going on)

dankrad · 2020-12-16T14:26:32Z

specs/phase1/beacon-chain.md

@@ -484,6 +485,10 @@ def process_shard_header(state: BeaconState,
    ))
 ```

+The length-and-degree proof works as follows. For a block B with length `l` (so `l` nonzero values in `[0...MAX_SAMPLES_PER_BLOCK-1]`), the length proof is supposed to be `(B / Z) * (r + X**(len(SETUP)-l))`, where `Z` is the minimal polynomial that is zero over `[l...MAX_SAMPLES_PER_BLOCK-1]` (see `SIZE_CHECK_POINTS` above). The goal is to ensure that a proof can only be constructed if (i) `B / Z` is itself non-fractional, meaning that `B` is a multiple of `Z`, and (ii) `deg(B) < MAX_SAMPLES_PER_BLOCK` (the block is not oversized).
+
+This is done by making the proof be a random linear combination of `B / Z` and `(B / Z) * (X**(len(SETUP)-l)`. The length proof will have the degree of `(B / Z) * X**(len(SETUP)-l)`, so `deg(B) - (MAX_SAMPLES_PER_BLOCK - l) + len(SETUP) - l`, simplified to `deg(B) - MAX_SAMPLES_PER_BLOCK + len(SETUP)`. Because it's only possible to generate proofs for polynomials with degree `< len(SETUP)`, it's this only possible to generate the proof if this expression is less than `len(SETUP)`, meaning that `deg(B)` must be strictly less than `MAX_SAMPLES_PER_BLOCK`.


The current code does not implement the random linear combination. But I believe it's not necessary anyway: We can simply make the combined max degree length proof be ( B * X **(len(SETUP) - l) ) / Z, which has a non-fractional solution only of B is 0 at the zeroes of Z.

So on the idea of getting rid of the zero check alltogether, and making it a pure degree check: I've got some python code here that interpolates a polynomial from only evaluations on a part of a subgroup:
https://github.com/ethereum/research/blob/master/polynomial_reconstruction/interpolate_polynomial_without_zeroes.py
Excluding the part that can be precomputed, it takes about 1.85s, and FFT in Go seems about 5x faster so an optimized version could do it in around 0.4s. So it feels like that's possible and I would think we should just do it?
This will lead to some savings in the proof computation, so we may well end up even or with some small savings. But we get a much nicer code for length verification which I think is worth it.
Also a nice effect is that the samples become more effective (=better coding rate) for smaller blocks.

[I implemented this as pure degree proofs for now, suggest we discuss the merits in this thread]

specs/phase1/beacon-chain.md

Co-authored-by: Radosław Kapka <rkapka@wp.pl> Co-authored-by: dankrad <dankrad@ethereum.org> Co-authored-by: terence tsao <terence@prysmaticlabs.com>

rkapka · 2021-01-01T07:47:43Z

specs/phase1/beacon-chain.md

+    shards = [
+        compute_shard_from_committee_index(state, slot, index)
+        for i in range()
+        state,
+        attestation.data.index,
+        attestation.data.slot
+    )


Suggested change

shards = [

compute_shard_from_committee_index(state, slot, index)

for i in range()

state,

attestation.data.index,

attestation.data.slot

)

rkapka · 2021-01-01T07:50:59Z

specs/phase1/beacon-chain.md

+                header.slot,
+                header.shard


Suggested change

header.slot,

header.shard

slot,

shard

rkapka · 2021-01-01T09:07:42Z

specs/phase1/beacon-chain.md

+            # The entire committee (and its balance)
+            full_committee = get_beacon_committee(state, slot, shard)
+            full_committee_balance = get_total_balance(state, full_committee)
+            if True not in [c.confirmed for c in candidates]:


I think it would make sense to include an assertion that the list is not empty because this is implied when setting the value of winning_index.

There is guaranteed to be one candidate per shard/slot combo due to the initialization that happens in reset_pending_headers

rkapka · 2021-01-01T15:38:39Z

specs/phase1/beacon-chain.md

-    # So how do we incentivise early custody key reveals now?
-    all_custody_secrets_revealed_epoch: Epoch  # to be initialized to FAR_FUTURE_EPOCH
+class BeaconBlock(phase0.BeaconBlock):
+    shard_headers: List[Signed[ShardHeader], MAX_SHARD_HEADERS]


A few comments here:

shard_headers is not set anywhere, thus assert len(body.shard_headers) <= (...) will always pass.

Since BeaconBlockBody contains a list of attestations, should shard_headers even exist? Shouldn't BeaconBlockBody.attestations hold headers within attestations themselves?

Even if we keep shard_headers, shouldn't the size of BeaconBlockBody.attestations be increased to cater for shards? Currently MAX_ATTESTATIONS = 128.

body.shard_headers will be set in the validator guide by the validator constructing the block

djrtwo

made a round of cleanups and minor fixes in 849837a

specs/phase1/beacon-chain.md

djrtwo · 2021-01-04T15:46:36Z

specs/phase1/beacon-chain.md


 ## Updated containers

 The following containers have updated definitions in Phase 1.

-### Extended `AttestationData`
+### `AttestationData`

 ```python
 class AttestationData(Container):


I have a feeling we won't be able to get away with the inheritance format due to the removal of fields but... we'll see

djrtwo · 2021-01-04T15:48:58Z

specs/phase1/beacon-chain.md

-    # So how do we incentivise early custody key reveals now?
-    all_custody_secrets_revealed_epoch: Epoch  # to be initialized to FAR_FUTURE_EPOCH
+class BeaconBlock(phase0.BeaconBlock):
+    shard_headers: List[Signed[ShardHeader], MAX_SHARD_HEADERS]


body.shard_headers will be set in the validator guide by the validator constructing the block

djrtwo · 2021-01-04T16:07:49Z

specs/phase1/beacon-chain.md

+        candidate_index = indices[compute_shuffled_index(i % total, total, seed)]
+        random_byte = hash(seed + uint_to_bytes(uint64(i // 32)))[i % 32]
+        effective_balance = state.validators[candidate_index].effective_balance
+        if effective_balance * MAX_RANDOM_BYTE >= MAX_EFFECTIVE_BALANCE * random_byte and effective_balance > min_effective_balance:


This will never exit if all validators' effective balance is less than min_effective_balance.

djrtwo · 2021-01-04T16:19:11Z

specs/phase1/beacon-chain.md

@@ -743,6 +414,9 @@ def process_operations(state: BeaconState, body: BeaconBlockBody) -> None:
    for_ops(body.attestations, process_attestation)
    for_ops(body.deposits, process_deposit)
    for_ops(body.voluntary_exits, process_voluntary_exit)
+    # Limit is dynamic based on active shard count
+    assert len(body.shard_headers) <= MAX_SHARD_HEADERS_PER_SHARD * get_active_shard_count(state, get_current_epoch(state))


I think we should just fix shard_count in this initial phase and get this check for free from typing

djrtwo · 2021-01-04T16:35:18Z

specs/phase1/beacon-chain.md

@@ -743,6 +414,9 @@ def process_operations(state: BeaconState, body: BeaconBlockBody) -> None:
    for_ops(body.attestations, process_attestation)
    for_ops(body.deposits, process_deposit)
    for_ops(body.voluntary_exits, process_voluntary_exit)
+    # Limit is dynamic based on active shard count
+    assert len(body.shard_headers) <= MAX_SHARD_HEADERS_PER_SHARD * get_active_shard_count(state, get_current_epoch(state))
+    for_ops(body.shard_headers, process_shard_header)


This needs to happen prior to process_attestation calls so that the pending headers for the newly added attestations already exist in state.

djrtwo · 2021-01-04T17:03:33Z

specs/phase1/beacon-chain.md

+            # The entire committee (and its balance)
+            full_committee = get_beacon_committee(state, slot, shard)
+            full_committee_balance = get_total_balance(state, full_committee)
+            if True not in [c.confirmed for c in candidates]:


There is guaranteed to be one candidate per shard/slot combo due to the initialization that happens in reset_pending_headers

specs/phase1/beacon-chain.md

nrryuya · 2021-02-23T07:47:29Z

specs/phase1/beacon-chain.md

+    )
+    previous_epoch_start_slot = compute_start_slot_at_epoch(get_previous_epoch(state))
+    for slot in range(previous_epoch_start_slot, previous_epoch_start_slot + SLOTS_PER_EPOCH):
+        for shard in range(SHARD_COUNT):


Should SHARD_COUNT be replaced with the value calculated by get_active_shard_count() ?

nrryuya · 2021-02-23T07:50:55Z

specs/phase1/beacon-chain.md

+                winning_index = [c.root for c in candidates].index(Root())
+            candidates[winning_index].confirmed = True
+    for slot_index in range(SLOTS_PER_EPOCH):
+        for shard in range(SHARD_COUNT):


Should SHARD_COUNT be replaced with the value calculated by get_active_shard_count() ?

nrryuya · 2021-02-23T09:23:12Z

specs/phase1/beacon-chain.md

+    previous_epoch_pending_shard_headers: List[PendingShardHeader, MAX_SHARD_HEADERS * SLOTS_PER_EPOCH]
+    current_epoch_pending_shard_headers: List[PendingShardHeader, MAX_SHARD_HEADERS * SLOTS_PER_EPOCH]
+    grandparent_epoch_confirmed_commitments: Vector[Vector[DataCommitment, SLOTS_PER_EPOCH], MAX_SHARDS]
+    shard_gasprice: uint64


Using Gwei instead of uint64 is clearer?

protolambda · 2021-03-30T14:04:44Z

#2254

Included this in above PR, we refactored the phase 1 spec into modules.

X-oss-byte

how about the block

vbuterin and others added 9 commits December 9, 2020 15:29

DAS phase 1

2cc458c

Combing... part 1

c80536c

Second pass

a63f68b

Fixed custom types

22dc5b2

Made confirmed headers a separate object

f192ffd

Abstracted (Kate commitment, length) into separate object

5f01e6c

Added dependencies to fork choice section

9f30f8a

Minor fixes

a354ded

(OOP SSZ) if the fields are updated, we need to redefine them when de…

ce6feee

…fining the new SSZ class with the new field classes

hwwhww force-pushed the vbuterin-patch-6 branch from ec62e15 to ce6feee Compare December 11, 2020 08:19

Revert: BeaconBlockHeader was not changed

f74f1f7

hwwhww reviewed Dec 11, 2020

View reviewed changes

specs/phase1/beacon-chain.md Show resolved Hide resolved

dankrad reviewed Dec 11, 2020

View reviewed changes

Update specs/phase1/beacon-chain.md

947c923

Co-authored-by: dankrad <dankrad@ethereum.org>

Grubson33421 approved these changes Dec 12, 2020

View reviewed changes

dankrad reviewed Dec 12, 2020

View reviewed changes

specs/phase1/beacon-chain.md Outdated Show resolved Hide resolved

specs/phase1/beacon-chain.md Outdated Show resolved Hide resolved

specs/phase1/beacon-chain.md Outdated Show resolved Hide resolved

vbuterin and others added 4 commits December 13, 2020 08:34

Update specs/phase1/beacon-chain.md

e38175f

Co-authored-by: dankrad <dankrad@ethereum.org>

Update specs/phase1/beacon-chain.md

fefb27e

Co-authored-by: dankrad <dankrad@ethereum.org>

Update specs/phase1/beacon-chain.md

1ce2039

Co-authored-by: dankrad <dankrad@ethereum.org>

pending_headers -> pending_shard_headers

569ca80

dankrad reviewed Dec 14, 2020

View reviewed changes

specs/phase1/beacon-chain.md Outdated Show resolved Hide resolved

dankrad reviewed Dec 14, 2020

View reviewed changes

specs/phase1/beacon-chain.md Outdated Show resolved Hide resolved

dankrad and others added 4 commits December 14, 2020 22:57

Some small fixes

99368ae

Delta minimum 1

939248c

Added size check to include the degree check

862ad2c

Add data availability coding rate as a constant (makes it more explic…

5b9f9b7

…it what's going on)

dankrad reviewed Dec 16, 2020

View reviewed changes

hwwhww mentioned this pull request Dec 17, 2020

Added data availability checks #1566

Closed

rkapka reviewed Dec 30, 2020

View reviewed changes

specs/phase1/beacon-chain.md Outdated Show resolved Hide resolved

rkapka reviewed Dec 30, 2020

View reviewed changes

specs/phase1/beacon-chain.md Show resolved Hide resolved

vbuterin commented Dec 31, 2020

View reviewed changes

specs/phase1/beacon-chain.md Show resolved Hide resolved

vbuterin commented Dec 31, 2020

View reviewed changes

specs/phase1/beacon-chain.md Show resolved Hide resolved

vbuterin commented Dec 31, 2020

View reviewed changes

specs/phase1/beacon-chain.md Outdated Show resolved Hide resolved

vbuterin and others added 2 commits December 31, 2020 09:35

Apply suggestions from code review

aad9c8d

Co-authored-by: Radosław Kapka <rkapka@wp.pl> Co-authored-by: dankrad <dankrad@ethereum.org> Co-authored-by: terence tsao <terence@prysmaticlabs.com>

Fixed index vs shard and beacon committee getting

4bfc910

ethereum deleted a comment from vbuterin Dec 31, 2020

rkapka reviewed Jan 1, 2021

View reviewed changes

protolambda mentioned this pull request Jan 1, 2021

DAS network/validator extension #2172

Closed

18 tasks

djrtwo force-pushed the vbuterin-patch-6 branch 2 times, most recently from 4205cf8 to b3c3c0a Compare January 4, 2021 18:28

djrtwo reviewed Jan 4, 2021

View reviewed changes

minor cleanups/fixes to shard data avail PR

849837a

djrtwo force-pushed the vbuterin-patch-6 branch from b3c3c0a to 849837a Compare January 4, 2021 18:30

hwwhww added the phase1 label Jan 7, 2021

lsankar4033 reviewed Jan 10, 2021

View reviewed changes

specs/phase1/beacon-chain.md Outdated Show resolved Hide resolved

nrryuya reviewed Jan 10, 2021

View reviewed changes

specs/phase1/beacon-chain.md Outdated Show resolved Hide resolved

djrtwo added 2 commits January 11, 2021 07:53

fix minor var name issue in DAS

f491be0

typo

18b50d6

nrryuya reviewed Feb 23, 2021

View reviewed changes

protolambda mentioned this pull request Mar 17, 2021

Modularize Phase1 into post-Merge features: Sharding, Custody Game, Data Availability Sampling #2254

Merged

18 tasks

protolambda closed this Mar 30, 2021

hwwhww added the scope:DAS Data Availability Sampling (DAS) label Mar 30, 2021

X-oss-byte reviewed Jun 24, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DAS phase 1 #2146

DAS phase 1 #2146

vbuterin commented Dec 9, 2020

dankrad Dec 14, 2020

vbuterin Dec 15, 2020

dankrad Dec 15, 2020

vbuterin Dec 16, 2020

dankrad Dec 16, 2020

dankrad Dec 19, 2020

dankrad Dec 14, 2020

vbuterin Dec 15, 2020

dankrad Dec 16, 2020

rkapka Jan 1, 2021

dankrad Dec 16, 2020

dankrad Dec 28, 2020 •

edited

Loading

dankrad Dec 28, 2020

rkapka Jan 1, 2021

rkapka Jan 1, 2021

rkapka Jan 1, 2021

djrtwo Jan 4, 2021

rkapka Jan 1, 2021 •

edited

Loading

djrtwo Jan 4, 2021

djrtwo left a comment •

edited

Loading

djrtwo Jan 4, 2021

djrtwo Jan 4, 2021

djrtwo Jan 4, 2021

djrtwo Jan 4, 2021

djrtwo Jan 4, 2021

djrtwo Jan 4, 2021

nrryuya Feb 23, 2021

nrryuya Feb 23, 2021

nrryuya Feb 23, 2021

protolambda commented Mar 30, 2021

X-oss-byte left a comment

DAS phase 1 #2146

DAS phase 1 #2146

Conversation

vbuterin commented Dec 9, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dankrad Dec 28, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rkapka Jan 1, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

djrtwo left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

protolambda commented Mar 30, 2021

X-oss-byte left a comment

Choose a reason for hiding this comment

dankrad Dec 28, 2020 •

edited

Loading

rkapka Jan 1, 2021 •

edited

Loading

djrtwo left a comment •

edited

Loading