Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Limit payload for request queue batch add #446

Closed
drobnikj opened this issue Nov 3, 2023 · 0 comments · Fixed by #489
Closed

Limit payload for request queue batch add #446

drobnikj opened this issue Nov 3, 2023 · 0 comments · Fixed by #489
Assignees
Labels
backend Issues related to the platform backend. bug Something isn't working. high priority Do this ASAP! This is for mission-critical work or work that blocks other teams in their work. t-platform Issues with this label are in the ownership of the platform team. validated Issues that are resolved and their solutions fulfill the acceptance criteria.

Comments

@drobnikj
Copy link
Member

drobnikj commented Nov 3, 2023

The maximum payload which Apify API handles is 9MB.
In some cases, the batch add requests payload overflow, and operation ends up with errors like:

2023-11-03T04:41:10.297Z ERROR Request batch insert failed
2023-11-03T04:41:10.299Z   ApifyApiError: The POST payload is too large (limit: 9437184 bytes, actual length: 9453568 bytes).(details: type=request-too-large)
2023-11-03T04:41:10.301Z     clientMethod: RequestQueueClient.batchAddRequests
2023-11-03T04:41:10.303Z     statusCode: 413
2023-11-03T04:41:10.305Z     type: request-too-large
2023-11-03T04:41:10.307Z     attempt: 1
2023-11-03T04:41:10.309Z     httpMethod: post
2023-11-03T04:41:10.311Z     path: /v2/request-queues/xxxxx/requests/batch
2023-11-03T04:41:10.313Z     stack:
2023-11-03T04:41:10.314Z       at makeRequest (/usr/src/app/node_modules/apify-client/dist/http_client.js:184:30)
2023-11-03T04:41:10.316Z       at runMicrotasks (<anonymous>)
2023-11-03T04:41:10.318Z       at processTicksAndRejections (node:internal/process/task_queues:96:5)
2023-11-03T04:41:10.320Z       at async RequestQueueClient._batchAddRequests (/usr/src/app/node_modules/apify-client/dist/resource_clients/request_queue.js:130:26)
2023-11-03T04:41:10.322Z       at async RequestQueueClient._batchAddRequestsWithRetries (/usr/src/app/node_modules/apify-client/dist/resource_clients/request_queue.js:153:34)

Let's chunk the payload into smaller batches to avoid these errors.

@drobnikj drobnikj added t-platform Issues with this label are in the ownership of the platform team. bug Something isn't working. high priority Do this ASAP! This is for mission-critical work or work that blocks other teams in their work. backend Issues related to the platform backend. labels Nov 3, 2023
@drobnikj drobnikj self-assigned this Nov 24, 2023
@drobnikj drobnikj added the validated Issues that are resolved and their solutions fulfill the acceptance criteria. label Feb 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend Issues related to the platform backend. bug Something isn't working. high priority Do this ASAP! This is for mission-critical work or work that blocks other teams in their work. t-platform Issues with this label are in the ownership of the platform team. validated Issues that are resolved and their solutions fulfill the acceptance criteria.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant