Test slog senders in a3p

[mhofman]

What is the Problem Being Solved?

In the past we're encountered issues with slog senders failing on some input (unknown or untested). Most recently #9569 / #8272.

While the slogfile, otel-trace and flight recorder are tested in the "deployment" integration test, that doesn't sufficiently cover all the possible slog events generated by a real chain (e.g. no smart wallet, no upgrade).

We should test the slog senders in a more realistic environment, such as a3p.

Description of the Design

Enable the different otel slog senders in a3p using environment variables, and SLOGSENDER_AGENT=process.

Enable failure of the chain process on slog sender failure.

Avoid leaving telemetry data in the final slog image somehow.

Security Considerations

None

Scaling Considerations

Slog processing can slowdown resource constrained systems like the CI runners

Test Plan

Somehow verify that slog senders are being used.

Upgrade Considerations

None, testing infra only