Unexpected error with client cluster using single reply queue

[itineric]

I am using single reply-to channel with spring rabbitmq to perform RPC. On the client side, everything is fine with basic tests. When duplicating the clients (setting clients up as a cluster), I am getting the following error:

Caused by: org.springframework.amqp.AmqpRejectAndDontRequeueException: Reply received after timeout
	at org.springframework.amqp.rabbit.core.RabbitTemplate.onMessage(RabbitTemplate.java:2729) ~[spring-rabbit.jar:3.0.9]

I looked at RabbitTemplate and I understand why I am getting this error.

Some details:
Client A is set up and listens to "custom-reply-to" channel, Client B is set up and listens to "custom-reply-to" channel
Client A sends a request, received by server and replied on "custom-reply-to" channel. Client B is listening to that channel and get the server's message before Client A does. Ok, thats fine, he will nack and requeue it... But we never get to the "nack" part.

In RabbitTemplate, this code makes the error occur.

PendingReply pendingReply = this.replyHolder.get(messageTag);
if (pendingReply == null) {

As I understand it, RabbitTemplate is looking with the given messageTag inside a Map where he stores the replies he is waiting for. But in my case, Client B is not waiting for this reply. Since it is Client A.

So getting the unexpected error and dont know how to manage my usecase.

[garyrussell]

When using named reply queues, each application instance needs to use its own reply queue; rather than using named queues, it is generally easier to use direct reply-to, where the broker configures a pseudo reply queue for each channel.

https://docs.spring.io/spring-amqp/docs/current/reference/html/#direct-reply-to

[itineric]

I undestand how it would/could work but I really need one reply queue. To simplify: this reply queue is monitored and when under load (and other checks), other clients are started.

As I understand it, it is not possible with Spring RabbitMQ.

[garyrussell]

That is correct; each instance needs its own reply queue; otherwise you would have multiple competing consumers on the single queue, with no guarantees that the reply will go to the request originator.

This has not a limitation of the Spring project; it's how RabbitMQ works; rejecting the message would not necessarily mean that it will be delivered to the other instance.

[itineric]

Isnt that the whole point of nack ?
It is working fine with RabbitMQ client (without spring). Having many clients with concurrency and only one reply queue. Need a fine tuning of "basicQos" but it work.

[garyrussell]

But it is highly likely it will be delivered to the wrong instance multiple times - wasting resources excessively, and the requesting instance might time out before it is redirected to that instance.

It is just not a good design.

[garyrussell]

Of course, you are completely free to subclass the RabbitTemplate, overriding onMessage, calling super.onMessage() and catch the AmqpRejectAndDontRequeueException and throw a different exception to requeue (nack) the message.