[BUG] Spring Cloud Stream Service Bus delivers a message sometimes twice

[mlohmeyer]

Describe the bug
Last Month we had 6000 cases (out of 5 Mio), where a Message gets delivered twice with Spring Cloud Stream Service Bus.

• The message stays for the Timeout in the Queue (2 Min, thats the minor problem)
• Then we process the message and complete it
• Then the message gets delivered again. (after another 2,8 mins)
• Then Spring Cloud Stream Azure is doing a renewMessageLock which fails, becuase the message is already completed
  But thats an internal exception, thus we process the message again. That the major problem.
  At the End complete fails, because the message is already processed.
• the error happens at arbitrary time. Not only on high Load on service Bus (Premium) and Container App.

Exception or Stack Trace
com.azure.messaging.servicebus.ServiceBusException: The lock supplied is invalid. Either the lock expired, or the message has already been removed from the queue. For more information please see ...

To Reproduce
unclear

Expected behavior
A Message get delivered only one time.

Screenshots

** Logs **

Logs from Service Bus Side (From Azure Support Ticket)
Message sent to queue | 0:12:01:01.529093
First delivery (msg lock) | 0:12:01:01.529272
Abandoned | 0:12:03:01.520244
Second delivery | 0:12:03:01.530253
Completed Successfully | 0:12:03:02.480113

what we can see in the application Insights Logs (100% of all logs are saved in Insight at that time)
first processing    on 2024-11-28T12:03:01.527Z (UTC)
complete         on 2024-11-28T12:03:02.366Z (UTC)
second processing on 2024-11-28T12:05:48.082Z (UTC) why?

**Setup **

• OS: Linux Docker Image
• Library/Libraries:
  spring-messaging-azure-4.15.0,
  spring-messaging-azure-servicebus-4.15.0
  spring-cloud-azure-stream-binder-servicebus-4.15.0
  spring-cloud-stream-4.0.3
  spring-boot-3.1.5
• Java version: 17
• App Server/Environment: Azure Container Apps
• Frameworks: Spring Boot

Information Checklist

• [ x ] Bug Description Added
• [ x ] Repro Steps Added
• [ x ] Setup information Added

[moarychan]

Hi @mlohmeyer ,

Thanks for using Spring Cloud Azure~

Please double check the versions you used, the version combination Spring Boot 3.1.5 + SCA Service Bus Binder 4.15.0 is not compatible; if you are using Spring Boot 3.1.5, SCA Service Bus Binder version should be 5.x. And could you provider a minimal project to reproduce the issue?

[akhudairymicrosoft]

Hi @moarychan , sample code and pom file were shared with you over email

[mlohmeyer]

Hi @moarychan we will upgrade to SCA Service Bus Binder 5.19.0, but it will lasts until mid on January because of cristmas holiday until we will get it productive.