Improve useage of Redis, Redis Streams, and other Redis components in inter-service communication

[robertbartel]

No description provided.

[robertbartel]

@christophertubbs I'm opening this to track this work. Feel free to split off subtasks as needed for more granular tracking. If you do, we should probably also change this issue to an Epic.

[christophertubbs]

The EvaluationService makes heavy use of redis pubsub and the evaluation logic itself may benefit from work distribution like that. Are there any other places where redis streams should be inserted?

[aaraney]

Are there any other places where redis streams should be inserted?

I would like to explore using redis streams as an event broker for the majority of service communication. To be more concrete, replacing websocket communication between services with publishing and subscribing to streams. I dont think streams is an optimal drop in replacement for our current communication, but that is not because of streams and instead the contents of the information we are sending between services at the moment.

[aaraney]

Not to get too far ahead of myself while also trying to lay down a little context, I dont think we will think about each service using the broker in the same way, meaning some services we may think of as passive consumers and others as active. The difference between an active and passive consumer comes down to if the consumer receives "commands" or "events" respectively. An example of what im calling a passive consumer could be the monitoring service. The monitoring service may not be send a "command" like, {"command": "LISTEN_FOR_JOB", "payload": { "id": "908c6bdb-50d3-48e1-99f2-00db82c0827e" }} but instead just listens on one or more streams and acts on the information it receives. In contrast, an example of an active consumer could be the data service where some other actor sends a command like CHECK_DATA, and the data service determines if data exists to satisfy the request and publishes an event with the outcome of the check.