Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

!feat(share/p2p/shrex-nd): rework shrex-nd to serve data in multiple messages #2444

Merged

Conversation

walldiss
Copy link
Member

@walldiss walldiss commented Jul 7, 2023

Overview

shrex-nd protocol is limited to send messages with max size of 1mb. PR modifies shrex-nd protocol to stream namespace data row by row. Next time we will have msg size problem after ods size > 2048.

Resolves #2419

@walldiss walldiss added area:p2p kind:feat Attached to feature PRs labels Jul 7, 2023
@walldiss walldiss self-assigned this Jul 7, 2023
@walldiss walldiss changed the title feat(share/p2p/shrex-nd): rework shrex-nd to serve data in multiple messages !feat(share/p2p/shrex-nd): rework shrex-nd to serve data in multiple messages Jul 10, 2023
@walldiss walldiss added the kind:break! Attached to breaking PRs label Jul 10, 2023
@codecov-commenter
Copy link

codecov-commenter commented Jul 10, 2023

Codecov Report

Merging #2444 (2154cc5) into main (eaa5835) will decrease coverage by 0.04%.
The diff coverage is 48.14%.

@@            Coverage Diff             @@
##             main    #2444      +/-   ##
==========================================
- Coverage   53.12%   53.08%   -0.04%     
==========================================
  Files         156      156              
  Lines        9907     9916       +9     
==========================================
+ Hits         5263     5264       +1     
- Misses       4188     4193       +5     
- Partials      456      459       +3     
Impacted Files Coverage Δ
share/getter.go 0.00% <0.00%> (ø)
share/p2p/shrexnd/params.go 12.50% <ø> (ø)
share/p2p/shrexnd/client.go 52.85% <46.93%> (+7.33%) ⬆️
share/p2p/shrexnd/server.go 52.00% <50.00%> (-1.03%) ⬇️

... and 6 files with indirect coverage changes

share/getters/shrex_test.go Outdated Show resolved Hide resolved
share/getters/shrex_test.go Outdated Show resolved Hide resolved
share/p2p/shrexnd/client.go Show resolved Hide resolved
share/p2p/shrexnd/server.go Outdated Show resolved Hide resolved
share/p2p/shrexnd/server.go Outdated Show resolved Hide resolved
share/p2p/shrexnd/server.go Show resolved Hide resolved
share/p2p/errors.go Outdated Show resolved Hide resolved
@walldiss walldiss requested a review from Wondertan July 10, 2023 15:30
@walldiss walldiss force-pushed the shrexnd_multiple_messages branch 2 times, most recently from 1f54ab8 to 2154cc5 Compare July 11, 2023 13:50
Copy link
Member

@renaynay renaynay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally we also get a test from the BlobModule-level to ensure this really fixes the issue e2e

share/p2p/metrics.go Outdated Show resolved Hide resolved
share/p2p/shrexnd/client.go Show resolved Hide resolved
}

return c.convertResponse(ctx, resp)
return c.convertStatusToErr(ctx, resp.Status)
}

// convertToNamespacedShares converts proto Rows to share.NamespacedShares
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

godoc

share/p2p/shrexnd/client.go Show resolved Hide resolved
share/p2p/shrexnd/client.go Show resolved Hide resolved
share/p2p/shrexnd/server.go Outdated Show resolved Hide resolved
share/p2p/shrexnd/server.go Show resolved Hide resolved
share/p2p/shrexnd/server.go Show resolved Hide resolved
share/getter.go Outdated Show resolved Hide resolved
share/getters/shrex_test.go Outdated Show resolved Hide resolved
Copy link
Member

@renaynay renaynay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are some issues with go mod for app I think.

@walldiss walldiss changed the base branch from main to v0.11.0-rc9 August 1, 2023 12:34
Wondertan
Wondertan previously approved these changes Aug 2, 2023
Wondertan
Wondertan previously approved these changes Aug 2, 2023
Wondertan
Wondertan previously approved these changes Aug 2, 2023
Wondertan
Wondertan previously approved these changes Aug 4, 2023
@walldiss walldiss dismissed Wondertan’s stale review August 4, 2023 11:34

The merge-base changed after approval.

@walldiss walldiss changed the base branch from v0.11.0-rc9 to release-11-rc9 August 4, 2023 14:29
@walldiss walldiss merged commit cf1b117 into celestiaorg:release-11-rc9 Aug 7, 2023
12 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:p2p kind:break! Attached to breaking PRs kind:feat Attached to feature PRs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

share/p2p/shrexnd: Support GetAll with > 1mb blob
4 participants