Add delay in the agent between each metric push

[knlambert]

Description

Currently:

• By default, envoy sends metrics every 3 4 seconds to the GRPC service from the agent.
• This delay is short, and not necessary since Ambassador cloud only stores data per minutes (last 30 minutes).

Changes:

• Added a 30s cooldown period before sending metrics again.
• Added some tests.
• Added a debug header for telepresence.

To do (need some help :o) )

• Expected version for this change ( Minor vs patch )
• Fix build having non related linting errors

Testing

• Tested in my cluster with a locally built version.
• Unit tests

Checklist

• I made sure to update CHANGELOG.md.
• This is unlikely to impact how Ambassador performs at scale.
• My change is adequately tested.

[LukeShu]

Can you please add an entry to docs/releaseNotes.yml (and then run make $PWD/CHANGELOG.md) summarizing these changes?

[LukeShu]

Some minor changes requested

• change request: Don't use context.TODO()
• change request: Don't use testify/suite
• suggestion: Come up with a better name than metricsRelayDeadline

But on the whole it looks good!

[knlambert]

Can you please add an entry to docs/releaseNotes.yml (and then run make $PWD/CHANGELOG.md) summarizing these changes?

Sure, under 2.3.0 ? or does it go in a 2.2.2 ?

[knlambert]

@LukeShu I have one question aswell about the metrics sink (that the agent is listening, and relaying).

Does it always push all the metrics ? Is there a risk I miss something by putting this 30s cooldown period ?

I see some differences between payloads, but only on some internal metrics which are not relevant to me.

[knlambert]

Rebased to fix conflict in the changelog file.

Changes made

[Alice-Lilith]

setting DCO to pass since @knlambert is with Ambassador Labs

[khussey]

@AliceProxy I guess this needs to be merged over to release/v2.3 in order to make the 2.3 release?

[Alice-Lilith]

@khussey I've already cherry-picked and merged these commits into the 2.3 release branch :)