request body too large from install-bundle / 400 from agoric publish

[dckc]

Describe the bug

I can't publish the 1.3M psm bundle.

The 836K mintHolder bundle works.

To Reproduce

I haven't tried to reproduce it, but this is what I did:

agoric-sdk/packages/vats$ yarn bundle-source --cache-json bundles/ src/mintHolder.js mintHolder

agoric-sdk/packages/vats$ agoric publish --node https://loadtest.rpc.agoric.net:443 bundles/bundle-mintHolder.json --chain-id agoric-emerynet-5-prime-4 --home ../cosmic-swingset/t1/8000
(Error#1)
Error#1: Bad status on response: 400

  at async publishBundleCosmos (packages/agoric-cli/src/publish.js:299:22)
  at async publishBundle (packages/agoric-cli/src/publish.js:455:5)
  at async publishMain (packages/agoric-cli/src/main-publish.js:59:26)
  at async Promise.all (index 0)
  at async main (packages/agoric-cli/src/main.js:368:5)

jittering agoric CLI deploy by 612ms
(Error#2)
Error#2: Bad status on response: 400

  at async publishBundleCosmos (packages/agoric-cli/src/publish.js:299:22)
  at async publishBundle (packages/agoric-cli/src/publish.js:455:5)
  at async publishMain (packages/agoric-cli/src/main-publish.js:59:26)
  at async Promise.all (index 0)
  at async main (packages/agoric-cli/src/main.js:368:5)

jittering agoric CLI deploy by 2882ms
(Error#3)

so we tried agd and got a little more diagnostics:

$ agd --node=https://loadtest.rpc.agoric.net:443 tx swingset install-bundle @bundles/bundle-mintHolder.json --from solo --chain-id agoric-emerynet-5-prime-4 -y -b sync
Error: RPC error -32600 - Invalid Request: error reading request body: http: request body too large

Expected behavior

the bundle gets published and visible from agoric follow :bundles

Platform Environment

• what OS are you using? what version of Node.js?
• is there anything special/unusual about your platform?

loadtest

• what version of the Agoric-SDK are you using? (run git describe --tags --always)

@agoric/cache@0.2.2-16-g438777d61 (a dev branch; changes are most likely unrelated)

cc @kriskowal

[JimLarson]

The rejected bundle is 750K.

[JimLarson]

Parameter in chain-config.js: RPC_MAX_BODY_BYTES. Should be ???.

[kriskowal]

I think multiple megabytes are sensible as long as we take payment based on content-length up front and drop the request if it exceeds its advertised content-length without a refund.

[dckc]

@michaelfig suggested that some network operation parameter outside the SDK was the limitation here. Whatever it was seems to have gone away. It's working today:

~/projects/agoric-sdk/packages/vats$ yarn bundle-source --cache-json bundles/ src/mintHolder.js mintHolder
yarn run v1.22.19
$ /home/connolly/projects/agoric-sdk/node_modules/.bin/bundle-source --cache-json bundles/ src/mintHolder.js mintHolder
ENOENT: no such file or directory, stat '/home/connolly/projects/agoric-sdk/packages/ERTP/src/types.js'
bundles/ add: mintHolder from src/mintHolder.js
bundles/ bundled 83 files in bundle-mintHolder.json at 2022-12-14T17:54:55.273Z
Done in 1.38s.

~/projects/agoric-sdk/packages/vats$ agoric publish --node https://devnet.rpc.agoric.net:443 bundles/bundle-mintHolder.json --chain-id agoricdev-16 --home ~/projects/devnet-deploy-test1/_agstate/agoric-servers/testnet-8000/
{"moduleFormat":"endoZipBase64Sha512","endoZipBase64Sha512":"392ae6656ae68bcfd8bd77a2a100ad4076e8b888808274796cce4c148fcb99d1f8b6c2f946bd6bc3e3c26c54c671c788890f3e9780464354fb2d30916e7be896"}

$ agd --node=https://devnet.rpc.agoric.net:443 query vstorage data bundles
value: '{"blockHeight":"89585","values":["{\"body\":\"{\\\"endoZipBase64Sha512\\\":\\\"392ae6656ae68bcfd8bd77a2a100ad4076e8b888808274796cce4c148fcb99d1f8b6c2f946bd6bc3e3c26c54c671c788890f3e9780464354fb2d30916e7be896\\\",\\\"error\\\":null,\\\"installed\\\":true}\",\"slots\":[]}"]}'

Then in a REPL, started with agoric start --reset -v testnet 8000 https://devnet.agoric.net/network-config:

command[4] E(home.zoe).installBundleID('b1-392ae6656ae68bcfd8bd77a2a100ad4076e8b888808274796cce4c148fcb99d1f8b6c2f946bd6bc3e3c26c54c671c788890f3e9780464354fb2d30916e7be896')
history[4] [Object Alleged: BundleIDInstallation]{}

command[8] E(zoe).startInstance(h[4], {}, {keyword: 'ZOT'})
history[8] {"adminFacet":[Object Alleged: adminFacet]{},"creatorFacet":[Object Alleged: ZOT mint]{},"creatorInvitation":undefined,"instance":[Object Alleged: InstanceHandle]{},"publicFacet":[Object Alleged: ZOT issuer]{}}
[12:40 PM]dckc | Agoric:
command[9] zot = {mint: h[8].creatorFacet}
history[9] {"mint":[Object Alleged: ZOT mint]{}}
command[10] zot.issuer = h[8].publicFacet
history[10] [Object Alleged: ZOT issuer]{}
command[11] E(zot.issuer).getBrand()
history[11] [Object Alleged: ZOT brand]{}
command[12] zot.brand=h[11]
history[12] [Object Alleged: ZOT brand]{}

command[14] E(zot.mint).mintPayment({brand: zot.brand, value: 123n})
history[14] [Object Alleged: ZOT payment]{}

[dckc]

This seems to be back... or something very much like it:

I'm getting Payload Too Large in an http function in node_modules/@cosmjs/tendermint-rpc/build/rpcclients/http.js when I run this under the debugger:

$ ls -sh ./bundles/bundle-psm.json
1.3M ./bundles/bundle-psm.json

$ agoric publish --home ~/.agoric/test-acct/ --node https://devnet.rpc.agoric.net:443 --chain-id agoricdev-17 ./bundles/bundle-psm.json
(Error#1)
Error#1: Bad status on response: 413

  at async publishBundleCosmos (packages/agoric-cli/src/publish.js:301:22)
  at async publishBundle (packages/agoric-cli/src/publish.js:470:3)
  at async publishMain (packages/agoric-cli/src/main-publish.js:59:26)
  at async main (packages/agoric-cli/src/main.js:395:5)

Using agd, it looks like...

$ ~/go/bin/agd tx swingset install-bundle @./bundles/bundle-psm.json --keyring-backend=test --from test-acct --node https://devnet.rpc.agoric.net:443 --chain-id agoricdev-17 --yes
Error: error unmarshalling: invalid character '<' looking for beginning of value

I suspect this blocks #6099.

The earlier fragmentation an reassembly work-around is based on ag-solo access, which we won't have in production.

[dckc]

tried it on a local node state-sync'd from devnet; no joy:

root@fa0b6c9bf9ff:/usr/local/src/agoric-sdk/packages/inter-protocol# agd tx swingset install-bundle @./bundles/bundle-psm.json --keyring-backend=test --from test-acct --chain-id=agoricdev-17 --yes
Error: RPC error -32600 - Invalid Request: error reading request body: http: request body too large

[dckc]

changed max_body_bytes = 1000000 to max_body_bytes = 1000000000 in config/config.toml. now we get:

# agd tx swingset install-bundle @./bundles/bundle-psm.json --keyring-backend=test --from test-acct --chain-id=agoricdev-17 --yes
code: 21
codespace: sdk
...
txhash: A09E3567FBBA4E197D1DAB4FC361464EA8E66A78B805837136467D2D7255F2F7

where 21 is...

	// ErrTxTooLarge defines an ABCI typed error where tx is too large.
	ErrTxTooLarge = errorsmod.Register(RootCodespace, 21, "tx too large")

aka max_tx_bytes from tendermint config

# NOTE: the max size of a tx transmitted over the network is {max_tx_bytes}.
max_tx_bytes = 1048576

[dckc]

On second thought, let's make a separate issue for the max_tx_bytes interaction: #7501.

[dckc]

JSON RPC is doubling the request size

I'm failing to install a bundle that's 0.607 MB compressed:

# agd tx swingset install-bundle @bundles/bundle-sell-concert-tickets.json --gas auto --keyring-backend test --chain-id agoriclocal --from agoric1s8lmczw37rz7qendxa2pc88qrlmhn4gsda8vn4 --broadcast-mode block --gas auto --gas-adjustment 1.4 --yes --output json
Error: RPC error -32600 - Invalid Request: error reading request body: http: request body too large

with help from @gibson042 , we looked at the traffic using tcpdump and realized that it's going over JSON RPC, which hex-encodes the protobuf message, doubling its size:

$ tcpdump -i any 'tcp dst port 26657' -w bundle-install.pcap
$ # run install-bundle
$ tcpdump -A -r bundle-install2.pcap
...
POST / HTTP/1.1
Host: localhost:26657
User-Agent: Go-http-client/1.1
Content-Length: 1290385
Content-Type: application/json

{"jsonrpc":"2.0","id":2,"method":"abci_query","params":{"data":"12FFAF270AA5AF270AA1AF270A...

A bundle that's 0.408 MB compressed works:

POST / HTTP/1.1
Host: localhost:26657
User-Agent: Go-http-client/1.1
Content-Length: 867661
Content-Type: application/json

{"jsonrpc":"2.0","id":2,"method":"abci_query","params":{"data":"12DDBC1A0A83BC1A0A

For reference, size details:

$ yarn test test/test-build-proposal.js 
yarn run v1.22.21
$ ava --verbose test/test-build-proposal.js

bundles bundle-postal-service.js valid: 202 files bundled at 2024-03-07T17:18:34.487Z with size 1687409
bundles bundle-sell-concert-tickets.js valid: 294 files bundled at 2024-03-07T17:18:32.583Z with size 2529869
  ✔ build:deployer package script generates compressed bundles less than 1MB (5.8s)
    ℹ {
        bundleName: 'postal-service',
        compressedSize: '0.408 MB',
        originallySize: '1.609 MB',
      }
    ℹ {
        bundleName: 'sell-concert-tickets',
        compressedSize: '0.607 MB',
        originallySize: '2.413 MB',
      }

context: I'm working on...

• tooling to deploy / start multiple contracts dapp-agoric-basics#15