Re-propagation of seen aggregated attestations dramatically increases bandwidth

[terencechain]

The re-propagation of seen aggregated attestations attestion_aggregate_proof was enabled in the consensus networking spec in Altair (See: rationale and pr)

Prysm implemented it in 9830 and included it in the release v2.0.3. Post release, we received multiple reports the outbound traffic quadrupled in certain situations for nodes then we root caused down to to the large increases in attestion_aggregate_proof re-propagation

Worth noting that Prysm may be the only client with re-propagation enabled, some other clients implementations are still in PR mode: lighthouse's pr

Opening this PR to document further findings, resolutions... etc

[prestonvanloon]

I observed a 100% (x2) increase in outbound network traffic on a small serving beacon-chain/validator client.
I did not observe quadruple increase. Testnets also did not reflect significant increase in outbound traffic, at least from the data we recorded. I'd like to verify that our testnet data sampling is accurately capturing p2p traffic, but I have no reason to believe it is not at this time.

[mohamedmansour]

I observed 2x CPU usage as well, is this related?

Network is 2x as well:

[prestonvanloon]

is this related?

Very likely.

v2.0.4 release candidate is 1d53fd2.
If you are up to it, please run from that commit. We are still testing / evaluating it with prod serving validators at this time, but believe CPU use is lower with the changes at 1d53fd2.

[mohamedmansour]

@prestonvanloon okay that fixed the cpu and network (on mainnet):

[prestonvanloon]

This is resolved in v2.0.4