[8.19] (backport #8248) feat: hybrid elastic agent invoke collector as subprocess

[mergify]

What does this PR do?

This PR introduces an architectural enhancement to how the Elastic Agent launches and supervises its internal OpenTelemetry (OTEL) collector:

• Refactors the OTEL manager to run the collector as a supervised subprocess rather than embedding it directly.
  - Adds a new edot-supervised sub-command to the Elastic Agent, enabling delegated startup of the OTEL collector with direct integration to the supervisor. After review comments: consolidated with the existing otel command
• Enables support for the healthcheckv2 extension in the OTEL configuration to enhance lifecycle and health signal handling from collector components.
• Improves test coverage and reliability by refactoring the otelmanager test logic to use time-based event checks, ensuring better guarantees.

This work sets the foundation for better fault isolation, process recovery, and alignment with existing process invocation model of elastic agent.

Why is it important?

Running the collector as a supervised subprocess provides improved fault isolation; a crash in the collector won't directly affect the main agent process.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

No immediate user impact is expected. The functionality remains the same as when running the collector directly inside the agent process. The new supervision model is internal and gated behind the edot-supervised command.

How to test this PR locally

mage unitTest

Related issues

N/A

---

This is an automatic backport of pull request #8248 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of c56581d has failed:

On branch mergify/bp/8.19/pr-8248
Your branch is up to date with 'origin/8.19'.

You are currently cherry-picking commit c56581d44.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   NOTICE-fips.txt
	modified:   NOTICE.txt
	modified:   go.mod
	modified:   go.sum
	modified:   internal/pkg/agent/application/application.go
	modified:   internal/pkg/agent/application/coordinator/coordinator_test.go
	modified:   internal/pkg/agent/cmd/otel.go
	modified:   internal/pkg/agent/cmd/otel_flags.go
	new file:   internal/pkg/agent/cmd/otel_test.go
	modified:   internal/pkg/otel/README.md
	new file:   internal/pkg/otel/agentprovider/buffer_provider.go
	new file:   internal/pkg/otel/agentprovider/buffer_provider_test.go
	modified:   internal/pkg/otel/agentprovider/provider.go
	modified:   internal/pkg/otel/agentprovider/provider_test.go
	new file:   internal/pkg/otel/agentprovider/scheme.go
	modified:   internal/pkg/otel/components.go
	new file:   internal/pkg/otel/manager/common.go
	new file:   internal/pkg/otel/manager/common_test.go
	new file:   internal/pkg/otel/manager/execution.go
	new file:   internal/pkg/otel/manager/execution_embedded.go
	new file:   internal/pkg/otel/manager/execution_subprocess.go
	modified:   internal/pkg/otel/manager/extension.go
	new file:   internal/pkg/otel/manager/healthcheck.go
	new file:   internal/pkg/otel/manager/healthcheck_test.go
	modified:   internal/pkg/otel/manager/manager_test.go
	new file:   internal/pkg/otel/manager/recovery_backoff.go
	new file:   internal/pkg/otel/manager/recovery_backoff_test.go
	new file:   internal/pkg/otel/manager/recovery_noop.go
	new file:   internal/pkg/otel/manager/testing/testing.go
	modified:   internal/pkg/otel/run.go
	modified:   internal/pkg/otel/translate/status.go
	modified:   pkg/component/runtime/log_writer.go
	modified:   pkg/core/process/process.go
	modified:   testing/integration/ess/otel_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   internal/pkg/otel/manager/manager.go
	both modified:   magefile.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elastic-sonarqube]

Quality Gate passed

Issues
6 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
74.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 141d8d2

Failed CI Steps

• fips:x86_64:sudo-false:fleet
• fips:arm64:sudo-false:fleet

History

• 💔 Build #23177 failed 2c91d4a

cc @pkoutsovasilis