Make the topic configurable for RedisMessageQueue

[masci]

No description provided.

[abdulhuq-cimulate]

@masci I wonder if this will address the issue I reported here.

TL;DR: When using RedisMQ with multiple replicas of the workflow, each workflow replica consumes and processes the same message from RedisMQ and as a result, the control plane ends up receiving duplicate responses from the workflow.
Using a single replica of SimpleMQ has resolved this problem but since it stores the MQ and Consumers in memory, we are planning on adding support for consumers to use a Redis KV store to make the SimpleMQ more resilient to pod restarts.

[masci]

@abdulhuq-cimulate that's a first step but it won't completely fix the issue: at the moment there's no policy around how messages are dispatched, some queues like Simple only supports one consumer per topic, others will happily relay the same message in the same topic to an arbitrary number of consumers (I guess this is your case).

My idea to fix it would be to introduce an explicit delivery policy, something like:

• DELIVER_ONCE that will deliver the message to at most one consumer, using round robin or random choice
• DELIVER_ALL that will deliver the message to all the registered consumers

For example we could set the delivery policy when creating the control plane, so it'll be unique for all the deployments and not dependent on the message queue backend (or happy to brainstorm other options).

[abdulhuq-cimulate]

@abdulhuq-cimulate that's a first step but it won't completely fix the issue: at the moment there's no policy around how messages are dispatched, some queues like Simple only supports one consumer per topic, others will happily relay the same message in the same topic to an arbitrary number of consumers (I guess this is your case).

My idea to fix it would be to introduce an explicit delivery policy, something like:

• DELIVER_ONCE that will deliver the message to at most one consumer, using round robin or random choice
• DELIVER_ALL that will deliver the message to all the registered consumers

For example we could set the delivery policy when creating the control plane, so it'll be unique for all the deployments and not dependent on the message queue backend (or happy to brainstorm other options).

Your fix makes sense.

I had a similar core idea but required introducing unncessary complexity by adding a separate MQ server that uses various existing MQs as backend that could take care of delivering messages only once using round robin. Alternatively, extend SimpleMQ to support different MQ backends.

Introducing a delivery policy for the messages in the control plane does make more sense.