feat: use square size params #1126

rahulghangas · 2022-12-15T07:22:08Z

Removes dependence of appconsts.DefaultMinSquareSize and appconsts.DefaultMaxSquareSize and instead fetches the relevant params from the blob keeper during process proposal

remove usage of default values
testing

app/prepare_proposal.go

cmwaters

Looks good. Just have a few trivial comments.

A slightly adjacent question I have with these upper and lower square size bounds is how are they initially calculated and how do we know when it's time to change them?

app/prepare_proposal.go

cmwaters · 2022-12-15T16:22:15Z

pkg/da/data_availability_header.go

+func ExtendShares(squareSize uint64, shares [][]byte, minSquareSize, maxSquareSize uint64) (*rsmt2d.ExtendedDataSquare, error) {
 // Check that square size is with range
- if squareSize < appconsts.DefaultMinSquareSize || squareSize > appconsts.DefaultMaxSquareSize {
+ if squareSize < minSquareSize || squareSize > maxSquareSize {
 return nil, fmt.Errorf(
 "invalid square size: min %d max %d provided %d",
- appconsts.DefaultMinSquareSize,
- appconsts.DefaultMaxSquareSize,
+ minSquareSize,
+ maxSquareSize,
 squareSize,
 )
 }


Just a thought here, but I feel like verification of the square size and extending the shares should be two separate functions. In PrepareProposal, estimateSquareSize already guarantees that the square size is within the expected range.

Also the comparison of squareSize*squareSize == len(shares) should probably be done in ComputeExtendedDataSquare.

pkg/da/data_availability_header.go

Co-authored-by: Rootul P <rootulp@gmail.com>

…data square

rootulp

[question] why aren't unit tests running on CI for this PR? I checked it out locally and I'm observing a test failure

--- FAIL: TestBlobInclusionCheck (0.03s)
    /Users/rootulp/git/rootulp/celestia/celestia-app/app/test/process_proposal_test.go:124: 
        	Error Trace:	/Users/rootulp/git/rootulp/celestia/celestia-app/app/test/process_proposal_test.go:124
        	Error:      	Not equal: 
        	            	expected: 1
        	            	actual  : 2
        	Test:       	TestBlobInclusionCheck
        	Messages:   	valid untouched data
FAIL
FAIL	github.com/celestiaorg/celestia-app/app/test	0.534s
FAIL

app/estimate_square_size.go

app/process_proposal.go

# Conflicts: # app/estimate_square_size.go # app/prepare_proposal.go # x/blob/types/payforblob.go

…een committed yet

rootulp

I don't have any blocking feedback but there is one unresolved blocking question on this PR: https://github.com/celestiaorg/celestia-app/pull/1126/files#r1049690184 so holding off on approval

app/test/process_proposal_test.go

rootulp · 2022-12-20T15:49:42Z

x/blob/keeper/params.go

+ if ctx.BlockHeader().Height < 1 {
+ return appconsts.DefaultMinSquareSize
+ }


[non blocking question] sorry I'm not familiar with defining genesis params. Is there a way to define the DefaultMinSquareSize param at chain genesis time so that we don't need this conditional?

If it isn't possible, why is a similar conditional not needed for the GasPerBlobByte param later in this file?

We ran into this today. Apparently the genesis params aren't set until the first block is committed. We added this check to min/max square size only because those two are queried in prepare/process proposal.

p.s. surprisingly it wasn't panicking (at the first block) on being unable to unmarshal nil byte slices to relevant values, but rather just blocking forever

I think it would be good to document this at least in the code, and then we might even want to document it elsewhere (but am unsure where)

Is it possible for someone to include a payForBlob in the first block of the chain? It seems unlikely but if yes, would GasPerBlobByte also need to be initialized?

that's a good point, I don't think so, as the params are used in many places, but I'm not sure of what is done to make that the case! perhaps we can do that process a bit earlier, so that it works for processproposal, which would almost certainly be a better solution

Apparently the genesis params aren't set until the first block is committed

This seems odd. You should be able to set params in genesis that can be used for the first block.

…imum square size

codecov-commenter · 2022-12-20T16:09:16Z

Codecov Report

Merging #1126 (0a296c1) into main (c16d6e7) will decrease coverage by 0.08%.
The diff coverage is 54.38%.

@@            Coverage Diff             @@
##             main    #1126      +/-   ##
==========================================
- Coverage   47.98%   47.90%   -0.09%     
==========================================
  Files          71       71              
  Lines        3995     4025      +30     
==========================================
+ Hits         1917     1928      +11     
- Misses       1909     1927      +18     
- Partials      169      170       +1

Impacted Files	Coverage Δ
app/prepare_proposal.go	`0.00% <0.00%> (ø)`
app/process_proposal.go	`0.00% <0.00%> (ø)`
app/estimate_square_size.go	`90.00% <62.50%> (-5.75%)`	⬇️
pkg/da/data_availability_header.go	`78.50% <100.00%> (-1.15%)`	⬇️
pkg/shares/share_splitting.go	`73.56% <100.00%> (+13.33%)`	⬆️
x/blob/keeper/params.go	`85.18% <100.00%> (-14.82%)`	⬇️
x/blob/types/payforblob.go	`80.35% <100.00%> (ø)`
pkg/shares/parse_sparse_shares.go	`68.96% <0.00%> (-14.12%)`	⬇️
pkg/shares/shares.go	`87.83% <0.00%> (-9.09%)`	⬇️
... and 7 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

x/blob/keeper/params.go

rahulghangas · 2022-12-21T07:22:30Z

I'm going to create an issue on the cosmos-sdk repository to get their opinion

cmwaters · 2022-12-21T12:09:32Z

x/blob/keeper/params.go

+ if ctx.BlockHeader().Height < 1 {
+ return appconsts.DefaultMinSquareSize
+ }


Apparently the genesis params aren't set until the first block is committed

This seems odd. You should be able to set params in genesis that can be used for the first block.

cmwaters · 2022-12-21T12:13:40Z

x/blob/keeper/params.go

@@ -21,12 +22,22 @@ func (k Keeper) SetParams(ctx sdk.Context, params types.Params) {

 // MinSquareSize returns the MinSquareSize param
 func (k Keeper) MinSquareSize(ctx sdk.Context) (res uint32) {
+ // use the default size for the first block so that we return a value on the
+ // first block in PrepareProposal
+ if ctx.BlockHeader().Height < 1 {


I'm pretty sure the first block is at height 1 not 0. Also we should be using something like InitialHeight as it's possible with Tendermint chains to start at any height.

Most importantly, I'm still not sure if this logic is necessary. At least I haven't seen this in other SDK modules.

I'm not sure either. However, with this logic removed, prepare proposal (where the params are fetched for the first time) blocks and using the cli to query the params panics with an error

chain logs

5:47PM INF Timed out dur=10000 height=1 module=consensus round=0 step=3 5:47PM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=0 numToDial=10 5:47PM INF No addresses to dial. Falling back to seeds module=pex 5:48PM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=0 numToDial=10 5:48PM INF No addresses to dial. Falling back to seeds module=pex

using celestia-appd query blob params results in

Error: rpc error: code = Unknown desc = UnmarshalJSON cannot decode empty bytes: panic

Interesting. Maybe you're right. I'd have thought the genesis params would be set at InitChain which comes before PrepareProposal but maybe I'm wrong.

What you could also do is add a log for when the params first get set in the Keeper and we can see if that comes before the first block gets proposed. Also wouldn't this panic you're seeing also happen in PrepareProposal?

evan-forbes · 2023-01-02T12:03:31Z

it looks like others have ran into this issue as well
cosmos/cosmos-sdk#14446

facundomedica · 2023-01-05T19:56:47Z

This might be a fix for this: cosmos/cosmos-sdk#14505 would love to get some more eyes on it.

The height in PrepareProposal was always currentHeight-1. Now that's fixed, PrepareProposal can't be called with height < 1.
The state from InitChain/Genesis wasn't accesible by PrepareProposal/ProcessProposal. That's also fixed, so you should be able to get params safely.

EDIT: Didn't mean "this" as the PR but as the problems pointed out in the comments. Feel free to ping me

cmwaters · 2023-01-18T13:34:19Z

Is there any update on this PR?

Can we either use the hack where we use the default size for the first block or we cherry-pick the commit that facundo did.

evan-forbes · 2023-01-18T13:57:36Z

I think the decision was to not pursue this change, so I think we can close this issue and PR.

IIRC, the reasoning was that while we want something like BlockSize (determined in MB), we don't want to expose square sizes as those are more complex internal params. We'd also still have a "HardMaxSquareSize", that we'd have to hardfork to change, so it seems a bit extra to have a blocksize param along with a max square size param along with a hard max square size

cmwaters · 2023-01-19T10:47:11Z

In that case can we look to either increase the current max square size from 128 to 256 i.e. 32MB square size or reduce DefaultMaxBlockBytes to 4MB. What I'm trying to get at is that the ceiling in terms of size should be something we control (i.e. MaxBlockBytes) and not something we don't (i.e. MaxSquareSize)

evan-forbes · 2023-01-29T02:51:04Z

closing since we're simply using the existing block size limits in tendermint

rahulghangas added 5 commits December 15, 2022 12:46

chore: remove unused constant

296d26c

feat: remove dependence on constant vals for min/max sqaure size

74253c3

feat: remove dependence on constant vals for min/max sqaure size

ef6ecb5

feat: pull params from blob module keeper

deb9035

test: refactor test to use new interface

e656e36

rahulghangas requested a review from evan-forbes as a code owner December 15, 2022 07:22

rahulghangas self-assigned this Dec 15, 2022

evan-forbes reviewed Dec 15, 2022

View reviewed changes

app/prepare_proposal.go Show resolved Hide resolved

app/prepare_proposal.go Show resolved Hide resolved

evan-forbes requested a review from rootulp December 15, 2022 14:16

evan-forbes added block validity rule consensus breaking modifies block validity rules in a way that will break consensus unless all nodes update their rules labels Dec 15, 2022

evan-forbes added this to the Incentivized Testnet milestone Dec 15, 2022

cmwaters reviewed Dec 15, 2022

View reviewed changes

rootulp reviewed Dec 15, 2022

View reviewed changes

pkg/da/data_availability_header.go Outdated Show resolved Hide resolved

rahulghangas and others added 6 commits December 19, 2022 17:24

Update pkg/da/data_availability_header.go

e3d33af

Co-authored-by: Rootul P <rootulp@gmail.com>

feat: add function for checking if square size is valid

4e4e4ad

chore: remove validity check for square size while creating extended …

35820d4

…data square

chore: remove extra parameters in call to ExtendShares

6d7f9f0

test: remove extra parameters in call to ExtendShares

4406e4a

feat: add square size check as a block validity rule

20a72aa

rahulghangas requested review from evan-forbes, cmwaters and rootulp December 19, 2022 13:33

test: check rejection on incorrect block size

2888fbe

rootulp reviewed Dec 19, 2022

View reviewed changes

app/estimate_square_size.go Outdated Show resolved Hide resolved

app/estimate_square_size.go Outdated Show resolved Hide resolved

app/process_proposal.go Show resolved Hide resolved

rahulghangas added 5 commits December 19, 2022 21:30

fix: incorrect logic for square size validity check

90b624b

chore: typo

bc82ef6

Merge branch 'main' into feat/use-squaresize-params

012666a

# Conflicts: # app/estimate_square_size.go # app/prepare_proposal.go # x/blob/types/payforblob.go

chore: convert uint32 to uint64

9a06093

test: refactor test to use updated function signature

eaabd6c

rahulghangas added 2 commits December 19, 2022 23:12

test: remove now invalid test

10b7e34

fix: return default values for min/max square size if not block has b…

076b9ad

…een committed yet

rahulghangas requested a review from rootulp December 20, 2022 13:39

rootulp reviewed Dec 20, 2022

View reviewed changes

rahulghangas added 2 commits December 20, 2022 21:31

test: test rejection of process proposal on square size less than min…

286eaaa

…imum square size

Merge branch 'main' into feat/use-squaresize-params

0fca546

evan-forbes reviewed Dec 20, 2022

View reviewed changes

x/blob/keeper/params.go Show resolved Hide resolved

chore: add documentation for edge case of fetching min/max square size

0a296c1

rahulghangas enabled auto-merge (squash) December 21, 2022 07:25

cmwaters reviewed Dec 21, 2022

View reviewed changes

evan-forbes modified the milestones: Incentivized Testnet, Incentivized Testnet Stretch Jan 3, 2023

rahulghangas disabled auto-merge January 5, 2023 08:23

evan-forbes closed this Jan 29, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: use square size params #1126

feat: use square size params #1126

rahulghangas commented Dec 15, 2022 •

edited

Loading

cmwaters left a comment

cmwaters Dec 15, 2022

rootulp left a comment

rootulp left a comment

rootulp Dec 20, 2022

rahulghangas Dec 20, 2022

evan-forbes Dec 20, 2022 •

edited

Loading

rootulp Dec 20, 2022

evan-forbes Dec 20, 2022

cmwaters Dec 21, 2022

codecov-commenter commented Dec 20, 2022 •

edited

Loading

rahulghangas commented Dec 21, 2022

cmwaters Dec 21, 2022

cmwaters Dec 21, 2022

rahulghangas Dec 21, 2022

rahulghangas Dec 21, 2022

cmwaters Dec 22, 2022

evan-forbes commented Jan 2, 2023

facundomedica commented Jan 5, 2023 •

edited

Loading

cmwaters commented Jan 18, 2023

evan-forbes commented Jan 18, 2023

cmwaters commented Jan 19, 2023

evan-forbes commented Jan 29, 2023

feat: use square size params #1126

feat: use square size params #1126

Conversation

rahulghangas commented Dec 15, 2022 • edited Loading

cmwaters left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rootulp left a comment

Choose a reason for hiding this comment

rootulp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

evan-forbes Dec 20, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented Dec 20, 2022 • edited Loading

Codecov Report

rahulghangas commented Dec 21, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

evan-forbes commented Jan 2, 2023

facundomedica commented Jan 5, 2023 • edited Loading

cmwaters commented Jan 18, 2023

evan-forbes commented Jan 18, 2023

cmwaters commented Jan 19, 2023

evan-forbes commented Jan 29, 2023

rahulghangas commented Dec 15, 2022 •

edited

Loading

evan-forbes Dec 20, 2022 •

edited

Loading

codecov-commenter commented Dec 20, 2022 •

edited

Loading

facundomedica commented Jan 5, 2023 •

edited

Loading