[beats receivers] The beatsauth extension should accept all HTTP transport settings and force use of our httpcommon.RoundTripper

[cmacknz]

• Relates [beats receivers] Test that translated Elasticsearch exporter configurations work with: self-signed certificates, mTLS, and proxies beats#45491
• Relates [beatreceivers] Integrate beatsauthextension #9257 (comment)

One of the biggest runtime risks for the Beats receivers project is deviation in how HTTP transport settings are applied. Differences in these areas run the risk of causing data ingestion failure and data loss by breaking connectivity with the output.

Fortunately, collector authenticator extensions allow us to inject our own http.RoundTripper and we can mitigate the risk of differences by inject the same round tripper that Beat HTTP clients use today.

This is the elastic-agent-libs httpcommon.RoundTripper which is configured from our standard set of transport settings:

type HTTPTransportSettings struct {
	// TLS provides ssl/tls setup settings
	TLS *tlscommon.Config `config:"ssl" yaml:"ssl,omitempty" json:"ssl,omitempty"`

	// Timeout configures the `(http.Transport).Timeout`.
	Timeout time.Duration `config:"timeout" yaml:"timeout,omitempty" json:"timeout,omitempty"`

	Proxy HTTPClientProxySettings `config:",inline" yaml:",inline"`

	IdleConnTimeout time.Duration `config:"idle_connection_timeout" yaml:"idle_connection_timeout,omitempty" json:"idle_connection_timeout,omitempty"`

The beatsauth extension should be updated to accept all of these settings and replace the collector's default RoundTripper with the one from httpcommon. Any one time initialization or file loading (e.g. certificate files) should happen in the extensions start method. This greatly minimizes the risk of configuration differences causing problems because our transport is the same one used by Beats.

When elastic-agent does configuration translation from elastic-agent.yml for beats receivers, it should always insert the beatsauth extension and pass it the complete set of transport settings accepted by httpcommon. We then guarantee that the HTTP client of each output is configured exactly the same as would have been for Beats, and equivalency with Beats is the main goal of the Beats receivers project.

There are some cons to this approach that we can accept initially:

• We will not be relying on the collector's RoundTripper and improved set of TLS configuration features (e.g. IncludeSystemCACertsPool) which we will want eventually. We should still aim to rely on these features, but rather than trying to use them to cover every possible use of Beats, we should switch over to configtls configuration on a case by case basis as we build up our equivalency testing.
• This approach will require two Elasticsearch exporters for Hybrid Agent, one configured explicitly and using configtls and the other our generated one using beatsauth extension.
  • We can over time eliminate this difference and converge on a single exporter, or alteratively move most use cases of the ES exporter to the OTLP exporter which would use configtls.

Acceptance Criteria

• Test(s) exists in elastic-agent proving the HTTP RoundTripper inserted in config translation is equal to the one from httpcommon and all HTTPTransport settings are handled in an equivalent way.
• Uses of tlsconfig in the oteltranslate package are removed and proven unnecessary https://github.com/elastic/beats/blob/b93bb0c7abc3ceeeca86aa8bde7dce4a08d85849/libbeat/otelbeat/oteltranslate/tls_otel.go#L93
• Configuration of proxy settings in ES output translation are removed as they are no longer necessary https://github.com/elastic/beats/blob/b93bb0c7abc3ceeeca86aa8bde7dce4a08d85849/libbeat/otelbeat/oteltranslate/outputs/elasticsearch/config_otel.go#L45-L63
• Configuration of idle_conn_timeout is handled by beatsauth extension and can be removed from config translation https://github.com/elastic/beats/blob/b93bb0c7abc3ceeeca86aa8bde7dce4a08d85849/libbeat/otelbeat/oteltranslate/outputs/elasticsearch/config_otel.go#L125
• The esexporter is audited for any configurations options that may no longer function or conflict with those from httpcommon.RoundTripper (compression for example should be re-checked).

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[cmacknz]

We will not be relying on the collector's RoundTripper and improved set of TLS configuration features (e.g. IncludeSystemCACertsPool) which we will want eventually. We should still aim to rely on these features, but rather than trying to use them to cover every possible use of Beats, we should switch over to configtls configuration on a case by case basis as we build up our equivalency testing.

I think the way we should approach this is to have some way to control which parameters we expect the beatsauth extension handle. Let's say we try to shift over to the collector's TLS settings and we are confident the translation is perfect, we would want a feature flag that allows users to revert back to the previously working configuration (e.g. beats auth extension handling everything).

We could technically use the beat receivers feature flag for this but if they are otherwise working we shouldn't make people give up the benefits (reduced memory and disk footprint) because we are tuning the connection settings.

[cmacknz]

Added #9803 as a sub-issue since this issue should fix that one.