[Meta] V2 Feature Architecture (8.5/8.6)

[blakerouse]

Overview

This is a meta issue to track all the items that need to be completed before the Elastic Agent is fully running and working with the new V2 architecture design. This issue tracks both the items that need to be done by the control plane team as well as other teams that interact with the Elastic Agent.

Control Plane

• Implement new v2 input specification
• Use the v2 input specification to generate the runtime model (aka. what should be running based on policy and v2 input specification)
• Refactor component binaries to be pre-extracted inside components directory with v2 specification
• Implement the v2 command runtime (runs each sub-process)
• Implement the v2 runtime manager (takes the computed runtime model and works to reconcile the model to the actual state)
• Implement the v2 server-side for the v2 control protocol
• Connect subprocess to Elastic Agent stdout/stderr
• Implement coordinator to manage config changes, variables (from composable inputs), and send it to the runtime manager.
• Refactor state tracking to use the coordinator as a central location of state management for the Elastic Agent.
• Use the new v2 runtime manager for standalone mode
• Use the new v2 runtime manager for Fleet mode
• Update the inspect command to provide inspection for the v2 runtime
• Update Fleet Server to use the new v2 control protocol
  • Owner: @blakerouse
• Use the new v2 runtime manager for Fleet Server Bootstrap
  • Owner: @blakerouse
• Display which integration is causing the agent to become unhealthy #100
• Add support for service based components (aka. Endpoint) [Elastic Agent V2]: Add support for service based components (aka. Endpoint) #1069
  • Owner: @aleksmaus
• Determine path forward for metrics: Tracking: current metrics monitoring with V2 #954
  • Owner: @michalpristas
• Determine path forward for diagnostics: [v2] Re-enable diagnostics for Elastic Agent and all components #1140
  • Owner: @blakerouse
• Fix running monitoring filebeat and metricbeat to send logs and metrics of running components to elasticsearch (Elastic Agent should aggregate the log from his subprocesses #221)
• Verify configuration reloading continues to work: V2 Configuration Reloading Checklist #1056
• Implement HasStrictExecPerms for Windows: V2 command work dir #1061 (comment)
• V2: Add integration tests for the agent v2 control protocol #1275

Known V2 issues

• V2: Agent doesn't stop cleanly on SIGINT: V2: Agent doesn't stop cleanly on SIGINT #995
• Fix agent shutdown on SIGINT #1258
• V2: Agent policy change is not applied properly and requires agent restart: V2: Agent policy change is not applied properly and requires agent restart #996
• V2: Agent starting multiple instances of filebeat or metricbeat and goes into tight restart loop for the beat process:
  V2: Agent starting multiple instances of filebeat or metricbeat and goes into tight restart loop for the beat process #997
• V2: Elastic Agent Install is broken. [V2]: Elastic Agent Install is broken. #1125
• V2: Removed download directory prevents the agent container from starting #1159
• V2: Intermittent panic on output reload beats#33125

Data Plane

• Migrate libbeat agent management to the agent V2 control protocol (Migrate libbeat agent management to the agent V2 control protocol beats#32078)
  • Owner: @fearful-symmetry
• Beats run under agent should start without reading a configuration file from disk. (Beats run under agent should start without reading a configuration file from disk. beats#31901)
  • Owner: @fearful-symmetry
• Agent V2: Automatically add agent info to Elasticsearch headers beats#32993
• Inject Beat gRPC stream identifiers without relying on agent specification file transformations beats#33135
• Agent V2: Check that a stream ID exists before generating an add_fields processor for it beats#33168

Security

• Implement V2 control protocol in Endpoint Security
  • Status: blocked on [Elastic Agent V2]: Add support for service based components (aka. Endpoint) #1069
  • Owner: @softengchick @brian-mckinney
• Implement V2 control protocol in Cloudbeat: v2 attempt cloudbeat#364
  • Status: Done, pending merge and more testing.
  • Owner: @eyalkraft
• Implement V2 control protocol in Osquerybeat:
  • Done in in Update Metricbeat, Filebeat, libbeat with elastic-agent V2 support beats#32673
  • Owner: @fearful-symmetry
• Implement V2 control protocol in Packetbeat and Auditbeat:
  • Status: blocked on Update Metricbeat, Filebeat, libbeat with elastic-agent V2 support beats#32673
  • Port Auditbeat and Packetbeat to the agent V2 control protocol beats#32786
  • Owner: @fearful-symmetry & @andrewkroh

Observability

• APM Server: Adapt to Elastic Agent v2 spec and libbeat changes apm-server#8958
  • Status: Done, pending merge.
  • Owner: @axw
• Heartbeat: [Heartbeat] States reloader and eslegclient beats#33405
  • Status: Done
  • Owner: @fearful-symmetry & @emilioalvap

Shipper Support

• Add v2 output specification for shipper support: Packaging for elastic agent elastic-agent-shipper#126
• Add shipper component runtime model calculation (when shipper enabled by environment) (includes calculating its configuration with processors from connected inputs)
  • [Meta][Shipper] Support the Elastic Agent Data Shipper #214
    • Owner: @blakerouse
    • Define a feature flag for enabling the shipper in the agent policy, defaulting to false. #217
    • [Meta][Feature] Start the shipper process(es) and configure inputs to write to them when the shipper feature flag is enabled. #219
    • Always provide the Elasticsearch output unit configuration to Heartbeat when using the shipper #1476
• Add shipper GRPC credential generation to runtime manager (connects inputs to shipper)

Packaging Shipper and Inputs

• Create build pipelines and packaging steps for new Elastic Agent Input and Shipper components ([META] Create build pipelines and packaging steps for all Elastic Agent subcomponents #592)
  • Status: in progress
  • Owner: @narph
  • https://github.com/elastic/release-eng/issues/268
  • https://github.com/elastic/release-eng/issues/283

[cmacknz]

Closing, everything remaining is now tracked as a bug against the release issue.