Getting ERR_FR_MAX_BODY_LENGTH_EXCEEDED with 12MB Feature

[metaskills]

Using the GitHub Action (https://github.com/devcontainers/action) which has the CLI v0.23.2, I get the following error when adding a 12MB file included with my internal feature. This happens during the putBlob function.

PUT new blob
Error [ERR_FR_MAX_BODY_LENGTH_EXCEEDED]: Request body larger than maxBodyLength limit

Full Actions Output with Error

2022-11-06T17:06:16.7087852Z [�[1m�[38;2;99;143;79m2 ms�[39m�[22m] @devcontainers/cli 0.23.2. Node.js v16.18.0. linux 5.15.0-1022-azure x64.
2022-11-06T17:06:16.7094737Z [�[1m�[38;2;99;143;79m2 ms�[39m�[22m] Packaging feature collection...
2022-11-06T17:06:16.7105561Z [�[1m�[38;2;99;143;79m5 ms�[39m�[22m] Processing feature: aws...
2022-11-06T17:06:16.7713950Z [�[1m�[38;2;99;143;79m65 ms�[39m�[22m] Processing feature: debug...
2022-11-06T17:06:16.7785925Z [�[1m�[38;2;99;143;79m73 ms�[39m�[22m] Processing feature: ssh...
2022-11-06T17:06:16.7846462Z [�[1m�[38;2;99;143;79m79 ms�[39m�[22m] Processing feature: vpn...
2022-11-06T17:06:16.8040996Z [�[1m�[38;2;99;143;79m98 ms�[39m�[22m] Packaged �[1m�[38;2;86;156;214m4�[39m�[22m features!
2022-11-06T17:06:16.8050121Z [�[1m�[38;2;99;143;79m99 ms�[39m�[22m] Processing feature: aws...
2022-11-06T17:06:16.8056120Z [�[1m�[38;2;99;143;79m100 ms�[39m�[22m] Fetching published versions...
2022-11-06T17:06:17.1509131Z [�[1m�[38;2;99;143;79m445 ms�[39m�[22m] Publishing versions: �[1m�[38;2;86;156;214m1�[39m�[22m,�[1m�[38;2;86;156;214m1.0�[39m�[22m,�[1m�[38;2;86;156;214m1.0.4�[39m�[22m,latest...
2022-11-06T17:06:17.4537718Z [�[1m�[38;2;99;143;79m747 ms�[39m�[22m] /tmp/features-output-1667754376705/devcontainer-feature-aws.tgz:  sha256:d34f42cc3870ad08ef6d1f3ff58986eee533ed5c3c07bbaeff80c69619261182 (size: �[1m�[38;2;86;156;214m12399616�[39m�[22m)
2022-11-06T17:06:17.4563512Z [�[1m�[38;2;99;143;79m748 ms�[39m�[22m] Computed Content-Digest ->  sha256:e89734603f16f351ad6e54c2d33a3021623bc26a503aa3d40aa197db90d1161a (size: �[1m�[38;2;86;156;214m64�[39m�[22m)
2022-11-06T17:06:18.2921524Z [�[1m�[38;2;99;143;79m1586 ms�[39m�[22m] PUT new blob -> 'sha256:d34f42cc3870ad08ef6d1f3ff58986eee533ed5c3c07bbaeff80c69619261182'
2022-11-06T17:06:18.3019794Z Error [ERR_FR_MAX_BODY_LENGTH_EXCEEDED]: Request body larger than maxBodyLength limit
2022-11-06T17:06:18.3021006Z     at RedirectableRequest.write (/usr/local/lib/node_modules/@devcontainers/cli/node_modules/follow-redirects/index.js:107:24)
2022-11-06T17:06:18.3022093Z     at /usr/local/lib/node_modules/@devcontainers/cli/dist/spec-utils/httpRequest.js:113:17
2022-11-06T17:06:18.3022520Z     at new Promise (<anonymous>)
2022-11-06T17:06:18.3023124Z     at requestResolveHeaders (/usr/local/lib/node_modules/@devcontainers/cli/dist/spec-utils/httpRequest.js:93:12)
2022-11-06T17:06:18.3023892Z     at putBlob (/usr/local/lib/node_modules/@devcontainers/cli/dist/spec-configuration/containerCollectionsOCIPush.js:212:74)
2022-11-06T17:06:18.3024732Z     at async pushOCIFeatureOrTemplate (/usr/local/lib/node_modules/@devcontainers/cli/dist/spec-configuration/containerCollectionsOCIPush.js:81:19)
2022-11-06T17:06:18.3025608Z     at async doPublishCommand (/usr/local/lib/node_modules/@devcontainers/cli/dist/spec-node/collectionCommonUtils/publishCommandImpl.js:71:14)
2022-11-06T17:06:18.3026705Z     at async featuresPublish (/usr/local/lib/node_modules/@devcontainers/cli/dist/spec-node/featuresCLI/publish.js:85:9)
2022-11-06T17:06:18.3027799Z     at async /usr/local/lib/node_modules/@devcontainers/cli/dist/spec-node/featuresCLI/publish.js:45:18 {
2022-11-06T17:06:18.3028315Z   code: 'ERR_FR_MAX_BODY_LENGTH_EXCEEDED'
2022-11-06T17:06:18.3028624Z }

Because there is no Lifecycle hooks for features (see devcontainers/spec#60) I opted to include this 12MB binary file in my feature. Certainly a ~12MB image is OK?

[joshspicer]

Ah yes, I think I know the issue. Our current implementation of the OCI distribution spec in the CLI currently doesn't page uploads.

The OCI spec linked above that method is not doing any chunking. I think we'd need to support pushing-a-blob-in-a-chunk to properly chunk upload.

cc/ @edgonmsft I think we knew this was a limitation but nobody had hit it yet. We should likely follow up with this, 12 MB seems reasonable to include in the uploaded Feature.

[DarthCoder117]

Just wanted to add that I'm also running into this issue trying to create a feature to install a custom GCC compiler for an embedded system.

[DarthCoder117]

Nothing close to being ready, but I forked the cli and started working on a patch myself.
Currently trying to figure out how to get tests running properly.
https://github.com/DarthCoder117/cli/pull/1/files

@joshspicer
The follow-redirects module uses a default max body size of 10MB, so I figure anything over 10MB makes sense to upload in chunks. I'd like to know hear maintainers opinions on whether or not this is a good solution.

[fightingsleep]

Hello. What is the current status on this issue? I'm running into the same limitation

[stewartfrancis]

@joshspicer @DarthCoder117 This doesn't appear to be fundamentally a chunking problem. From my investigation, the max request body size is a restriction that's solely imposed by the http client. The problem seems to be that follow-redirects imposes a 10MB default limit to maxRequestBody as documented here: https://www.npmjs.com/package/follow-redirects

I was able to set this limit higher (e.g. 100MB) in src/spec-utils/httpRequest.ts by adding maxBodyLength to requestOptions. I was then able to publish larger features (obviously up to 100MB in my case).

I'm happy to try and put a PR together, but I'd need some guidance over what a sensible limit to maxBodyLength should be, and whether or not this needs to be user-configurable. Seems like this is quite easy to achieve vs implementing chunking in the client (which would be an alternate solution).

[joshspicer]

Thanks for investigating! I don't think I have a strong opinion on sensible upper limit, although increasing by a factor of 10 for now (to the 100 MB) you've tested seems reasonable to me. Layers in container images are often larger than 100 MB, so that seems just as safe as the 10 MB "limit" currently imposed. We'd definitely accept a PR here, thank you!

cc/ @chrmarti thoughts?

[chrmarti]

Sounds good. I guess the affected request shouldn't be redirected anyway (the data transfer would be done twice when the first request replies with a redirect?).