Graceful degradation of UDP client senders on EMSGSIZE

[vprithvi]

When attempting to send large spans, some clients hit a UDP's EMSGSIZE, which means that the data being sent is larger than the maximum UDP packet size.

Debugging this error is difficult because clients only log the error and increment a metric.

I propose that when clients get a EMSGSIZE back, they attempt to resend the Span after stripping out tags and logs, and inserting a UDP_SPAN_TOO_BIG tag, or something similar. I expect this to increase the success of sending a Span because the only variable sized fields are tags, logs, and operationName.

[jpkrohling]

Perhaps a good first step is to add a WARN log entry + increase a counter in a specific (new?) metric.

[yurishkuro]

Maybe a better approach is for clients to run a form of a health check on the agent, e.g. submit a large packet of garbage data over UDP so that the agent can register the new client uuid, and then query the http config endpoint to see if the agent received such packet successfully. If it didn't, then the client can either start complaining loudly, or drop all spans and keep checking the health endpoint, possibly trying smaller max packet sizes.

[jpkrohling]

I think we did talk about having something like that for the connectivity test. Perhaps as part of #1054?

[yurishkuro]

Per #3362, we're sunsetting Jaeger clients.