[SOL-38491] queueMaxMsgRedelivery not working

[FWinkler79]

My use case is the following:

• A producer sends a message
• The consumer processes the message (by calling another service) and in case of an error, no (Spring) retry should take place, instead the message should immediately be re-queued and re-delivered.
• In case the message was re-queued 3 times and could not be processed properly, it should no longer be re-queued and re-delivered.

I am using the following configuration:

spring:
  cloud:
    stream:
      default:
        group: defaultConsumers
        consumer:
          concurrency: 3
      bindings:
        jobTriggers:  
          group: jobTriggerConsumers 
          # Effectively disable internal retry of failed message processing attempts.  
          # Instead we will re-queue them in the solace broker.
          # See: https://cloud.spring.io/spring-cloud-static/spring-cloud-stream/2.2.1.RELEASE/spring-cloud-stream.html#_re_queue_failed_messages
          consumer:
            concurrency: 3
            max-attempts: 1
      # Make sure failed messages are requeued in the solace broker. 
      # See: https://github.com/SolaceProducts/solace-spring-cloud/tree/master/solace-spring-cloud-starters/solace-spring-cloud-stream-starter#failed-message-error-handling 
      solace: 
        bindings:
          jobTriggers:
            consumer:
              dmq-max-msg-redelivery: 3   # has no effect
              queue-max-msg-redelivery: 3 # has no effect
              requeue-rejected: true

with the following @StreamListener code:

@StreamListener(JobTriggerEventConsumerBinding.INPUT)
protected void onJobTriggerEvent(org.springframework.messaging.Message<JobExecutionTriggerEvent> message, 
                                 MessageHeaders headers,
                                 @Validated JobExecutionTriggerEvent event) throws InterruptedException { 
   throw new RuntimeException("This error should re-queue the message. After three re-queue attempts no more re-queuing should happen");
}

Expected behaviour: Message processing should not be retried (i.e. using RetryTemplate), but the message should be re-queued and immediately re-delivered to the @StreamListener. After 3 re-delivery attempts (spring.cloud.stream.solace.bindings.jobTriggers.consumer.queue-max-msg-redelivery: 3) the message should not be re-queued and no re-delivery should take place.

Observed behaviour: The message gets re-delivered indefinitely. This is independent of whether queue-max-msg-redelivery is used or queueMaxMsgRedelivery. Also the dmq-max-msg-redelivery does not have any effect here.

This is crucial defect, as it leaves the application without any chance to recover from failure scenarios, if the application wants to use message re-delivery as described above.

[Mrc0113]

Thanks for opening this ticket @FWinkler79! You did indeed find a bug.

It looks like the queue-max-msg-redelivery setting works properly. If you look at the created queue (make sure it's not already created before running your app) you can see that the "Maximum Redelivery Count" matches the value placed in this config.

The bug seems to be in the re-queue logic that is enabled when requeue-rejected: true. Instead of unbinding the flow and essentially "nacking" the message so the redelivery count gets incremented it looks like the binder is actually republishing a new message which is a bug. You can see that here:

solace-spring-cloud/solace-spring-cloud-stream-binder/solace-spring-cloud-stream-binder-core/src/main/java/com/solace/spring/cloud/stream/binder/util/SolaceErrorMessageHandler.java

Line 99 in 967921c

private void requeue(XMLMessage rawMessage) {

Heads up @Nephery! We can discuss this tomorrow as well :)

@FWinkler79 as an FYI, I ran out of time today but I'm hoping to take a closer look at your other ticket tomorrow.

[Nephery]

Closed with #75