Reducing Elastic Agent memory and cpu footprint

[jlind23]

This task is a place holder to track all ongoing effort and issues related to reducing Elastic Agent cpu and memory footprint.

Some of the ideas are:

• Having a single data shipper per multiple processes
• Reducing the number of processors that are duplicated on every input
• https://github.com/elastic/ingest-dev/issues/1151
• Investigate changing the pipeline order to improve performance elastic-agent-shipper#44
• Automatically set GOMAXPROCS elastic-agent-shipper#55

[jlind23]

@pierrehilbert @cmacknz Is there any issue that should be linked in this meta? What about the gomaxproc one?

[cmacknz]

• The shipper will reduce the memory footprint by moving from a queue per input type in the policy to a queue per output. This is the biggest impact I can think of.

These don't let us change the minimum requirement, but would give us a way to control the maximum:

• Allowing us to specify the memory queue size in bytes instead of events would give us a way to limit memory. Related to https://github.com/elastic/ingest-dev/issues/1151 and Investigate changing the pipeline order to improve performance elastic-agent-shipper#44 but no concrete implementation issue yet.
• The Go Max procs / agent CPU limit issue would give us a way to limit the maximum CPU usage.

[jlind23]

Countless achievements have been done on this hence closing for now.
cc @ycombinator