Ambassador Agent fails to report snapshot to Ambassador Cloud on "large" snapshots

[acookin]

In larger installations of ambassador, the agent hits the default grpc max message size when trying to relay the snapshot to the cloud.

Error:

time="2021-07-01 15:22:33" level=warning msg="failed to report: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (8884732 vs. 4194304)" 

We should probably stream the snapshot in chunks (we could bump the default message size, but that's a bandaid, not a solution.)

This error was reported by an ambassador cloud user in slack.

[andrewnazarov]

I was that reporter. Feel free to ping me here if you need some additional information:)

[ankit-ls]

@acookin I am observing the same issue consistently.

{"log":"time=\"2021-07-20 07:16:01\" level=warning msg=\"failed to report: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (11909674 vs. 4194304)\" func=\"github.com/datawire/ambassador/pkg/agent.(*Agent).MaybeReport.func1.1\" file=\"/go/pkg/agent/agent.go:474\" CMD=agent PID=1\n","stream":"stderr","time":"2021-07-20T07:16:01.996242695Z"}

Any timeline for a fix on this? We use helm charts to install the agent, so a parameter to increase the message size here (potentially as an agent.env[0] variable) should also be sufficient for us, as a short term fix.

[acookin]

@ankit-ls we're working on this issue right now, will update here when we've solved it