Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pagination to avoid Blobbers being unresponsive #228

Closed
Kishan-Dhakan opened this issue Jun 27, 2021 · 1 comment
Closed

Add pagination to avoid Blobbers being unresponsive #228

Kishan-Dhakan opened this issue Jun 27, 2021 · 1 comment

Comments

@Kishan-Dhakan
Copy link
Contributor

Kishan-Dhakan commented Jun 27, 2021

Problem:
The zboxcli uses commands like list, list-all and the other CLIs in progress like s3 and nfs will also have list operations. There is currently no support for pagination, so the entire list is given as a respondse. This is fine if there are smaller number of files but we need to consider that an allocation can contain thousands of such files. For about 5 directories and 5 files the response size was about 60KB so for large number of files it will be large sized response making blobber busy to serve request for certain amount of time as metadata can be of for example; 20MB which obviously stalls the Blobber.
And this is just for single allocation. Blobbers however are not confined to single allocation and there can be multitude of clients requests.

Techincal description:
These repos currently use the GoSDK function NewListRequest in gosdk/zboxcore/zboxutil/http.go. This makes an GET request to the endpoint /v1/file/list/ which only takes in two params, path_hash and auth_token . It would be good if this endpoint allows a param for pagination (offset or limit or something similar) which the GoSDK can levarage.

Follows issue 117

@moldis
Copy link
Contributor

moldis commented Oct 25, 2021

PR: #325

@moldis moldis closed this as completed Oct 25, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants