Create a skeleton shipper executable

[cmacknz]

Create a skeleton shipper executable that:

• Logs to stdout per the control plane pipeline specification.
• Starts a gRPC server when run, that exposes the event gRPC service.
• Can start an HTTP server based on a command line flag that:
  • Exposes the /debug/vars endpoint for metrics (or similar, like the beats /stats endpoint).
  • Exposes the /debug/pprof endpoint for profiling.
• Consumes a configuration file that can configure the gRPC and HTTP interfaces at a minimum. The configuration file will be a subsection of the Elastic agent policy and should use the same format.

Attempt to reuse as much of the agent code as we can for these tasks: elastic/elastic-agent#276

[fearful-symmetry]

Okay, finally digging into this and how it should work. I'm kind of working on the assumption that the first PR/commit will literally be a gRPC skeleton with some basic server code. However, I'm not quite sure what the interface between the gRPC server and the memory queue is supposed to look like.

[cmacknz]

Let's not worry about the interface to the memory queue yet. Let's just get an executable with a gRPC server running.

We can sort out how the memory queue connects gRPC interface as part of #7.

Once we have all the gRPC scaffolding done, you can move onto #11 and leave the output pipeline as part of #7

[fearful-symmetry]

So, now that I have a basic shipper up, I'm looking at the http/pprof requirement here. The elastic-agent actually has a bunch of the init/management code in internal/pkg/agent/cmd/run.go. I'm kind of wondering if we should library-ize this existing code and the associated config objects, then import them back into agent and the shipper. Thoughts @kvch @cmacknz ?

[cmacknz]

I'm kind of wondering if we should library-ize this existing code and the associated config objects, then import them back into agent and the shipper. Thoughts @kvch @cmacknz ?

Not a bad idea, that code already has the signal handling setup and will probably use the agent policy config format (what the shipper should use). Might be a bit of work though, I'd ask the control plane team what they think and if it's a good idea create a separate issue to do it.

[fearful-symmetry]

@elastic/elastic-agent-control-plane Do we have any opinions on this?

So, now that I have a basic shipper up, I'm looking at the http/pprof requirement here. The elastic-agent actually has a bunch of the init/management code in internal/pkg/agent/cmd/run.go. I'm kind of wondering if we should library-ize this existing code and the associated config objects, then import them back into agent and the shipper.

[ph]

Thanks for the ping, this is good timing indeed, last Friday. I was looking at our run and how we are bootstrapping our system in Agent. I am not a super fan of how it's currently made. We have indeed inherited some of the beats cruft and some of our own.. so I've started this experimentation. https://github.com/ph/vif/blob/main/cmd/vif/main.go I wanted to facilitate our development of the v2 and be able to mock inputs/shippers.

• I wanted to simplify the bootstrap.
• Separate the signal handling and the execution so we can test the initialization of the application without signals.
• Having a more sane logger started early and being able to change log level on the fly.
• Having a more well-defined startup: Metric, Httprof, and other subsystem.
• Making it composable as a library.

I think having a library that makes little assumption would make a lot of sense.

• Shipper could use it
• When we refactor the input it could use it.
• We can refactor the agent to use it.

@fearful-symmetry creates an issue I would be happy to drop a few things on our side.

[ph]

Note: the code was just an experiment, take it with a grain of salt. :)

[fearful-symmetry]

I think once #29 is merged, we should be able to close this?

[cmacknz]

The only additional thing I think we are missing is signal handling for graceful shutdown. This is provided in the libbeat service package for example: https://github.com/elastic/beats/blob/main/libbeat/service/service.go#L42

We could do that under this issue or make a separate issue for it.

[fearful-symmetry]

Ah, I knew I was forgetting something.

[fearful-symmetry]

The merged queue monitor also includes a signal handler for shutdown, so we should be good here?

[cmacknz]

Yes, closing. Thanks!