Add broker-side job stream service API

[npepinpe]

Description

Job workers/streams will be managed in-memory by the broker. Since workers are always interested in all partitions, this can be a long living, singleton instance on the broker side.

The external job activator will be able to query this service to activate jobs.

In the prototype, I used SWIM to register/unregister workers/streams. However, I wasn't differentiating between workers other than by type. Since we want to keep the same API and treat each worker instance differently based on their properties, we cannot do this here.

There will be a new API in the broker which can receive commands from the gateway (i.e. using the protocol), with only two RPCs:

• AddWorker
• RemoveWorker
• RemoveWorkers

An AddWorker request should contain the job type, worker's activation properties, and the gateway/recipient member ID where a job would be pushed.

Workers should be uniquely identifiable by their gateway (i.e. push recipient member ID) and their activation properties. Two workers with the same recipient and the same properties are logically equivalent and do not need to be registered multiple times.

Workers can be removed on demand via the RemoveWorker API, using their recipient and properties to identify them.

Note As this can be somewhat verbose, we may want to introduce an ID or something to make it easier to manage.

A RemoveWorkers request is received from a gateway (with its member ID as param), and all workers associated with it are removed. If a gateway dies (as detected via SWIM), the same should happen.

Note that we are not implementing pushing jobs here yet, just managing the lifecycle of workers. With this done, we can then provide a dummy external job activator implementation (unusable for production of course) which can activate jobs. We can then test that the jobs are activated with the right properties, and variables are collected, etc.

I would recommend further breaking down this issue, or are least completing it in multiple PRs.

[npepinpe]

Additionally:

• Add registry metrics
• Implement GatewayStreamer and GatewayStream which pushes to a specific topic
• Emit metric on push

[npepinpe]

@deepthidevaki once you've merged the last PR, as discussed, please add a gauge for the registered stream count, trace logging of every job pushed, and debug logging of the stream request handler (adding, removing, etc.)