Add package and API skeleton for internal queue

[faec]

This is the first pass of integrating the libbeat queue into the shipper server. It creates the package namespace and structures and passes through the initialization / shutdown process, but doesn't yet use a real queue internally (that is waiting on the libbeat-side api changes to be merged in elastic/beats#31699). But this creates a skeleton and moves things to the right places so we can keep building around it without version skew from the draft PRs.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-05-26T16:56:20.698+0000

• Duration: 8 min 27 sec

Test stats 🧪

Test	Results
Failed	0
Passed	6
Skipped	0
Total	6

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[cmacknz]

Looks good as a starting point. Tests are coming later I assume?

[faec]

Looks good as a starting point. Tests are coming later I assume?

Yeah, I just wanted to lock in the structure that isn't dependent on the outstanding libbeat api change, I'm expecting some unit tests when that gets merged and probably additional end-to-endish ones following that to test the whole process.

[kvch]

Why are you using this? Is this a temporary import and going to be removed in a later relase?
I was under the impression that we are trying to avoid importing from beats in our new repositories. If we want to reuse something from beats we should either move it to elastic-agent-libs or copy it to the new repo where it is used.

[cmacknz]

The shipper is initially going to be built out of the beats outputs, queues, and the code that glues them together. That is a large chunk of code to move that at this point would just slow down development. I would like us to prioritize getting a working shipper prototype over the code reorganization right now.

I am in favour of moving all that code into elastic-agent-libs afterwards or perhaps in the background. That will make it clear that the code is used outside of just beats now.

I will create an issue to migrate the code we end up using out of beats, which I assume will also be easier once we know what pieces we need to reuse.

[faec]

Yeah, there is still more pending work to clean up and modularize the queue and I expect that as that progresses it will probably migrate to elastic-agent-libs, it just wasn't part of the first draft.

[cmacknz]

#43