JMS Integration with Tibco causes deadlock while using DefaultMessageListenerContainer [SPR-15738]

[spring-projects-issues]

Anup Tagalpallewar opened SPR-15738 and commented

I am facing this issue while using
Spring-jms version 4.1.6
tibjms version 6.3 (Tibco jar)

When using a DefaultMessageListenerContainer, if there is an exception on established connection we intermittently get this deadlock issue.

Please find following stack trace for more information.

"clientResponseListener-16" - Thread t@242

java.lang.Thread.State: BLOCKED
at org.springframework.jms.connection.SingleConnectionFactory$SharedConnectionInvocationHandler.localStop(SingleConnectionFactory.java:640)

• waiting to lock <1768ffa4> (a java.lang.Object) owned by "TIBCO EMS TCPLink Reader (Server-999487)" t@241
  at org.springframework.jms.connection.SingleConnectionFactory$SharedConnectionInvocationHandler.invoke(SingleConnectionFactory.java:573)
  at com.sun.proxy.$Proxy54.stop(Unknown Source)
  at org.springframework.jms.connection.SingleConnectionFactory$SharedConnectionInvocationHandler.localStop(SingleConnectionFactory.java:644)
• locked <1ea19285> (a java.lang.Object)
  at org.springframework.jms.connection.SingleConnectionFactory$SharedConnectionInvocationHandler.invoke(SingleConnectionFactory.java:577)
  at com.sun.proxy.$Proxy54.close(Unknown Source)
  at org.springframework.jms.connection.ConnectionFactoryUtils.releaseConnection(ConnectionFactoryUtils.java:80)
  at org.springframework.jms.listener.AbstractJmsListeningContainer.refreshSharedConnection(AbstractJmsListeningContainer.java:395)
• locked <78213c8c> (a java.lang.Object)
  at org.springframework.jms.listener.DefaultMessageListenerContainer.refreshConnectionUntilSuccessful(DefaultMessageListenerContainer.java:909)
  at org.springframework.jms.listener.DefaultMessageListenerContainer.recoverAfterListenerSetupFailure(DefaultMessageListenerContainer.java:884)
  at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1055)
• locked <5f74909f> (a java.lang.Object)
  at java.lang.Thread.run(Thread.java:745)

Locked ownable synchronizers:

• None

  "TIBCO EMS TCPLink Reader (Server-999487)" - Thread t@241

java.lang.Thread.State: BLOCKED
at org.springframework.jms.connection.SingleConnectionFactory$AggregatedExceptionListener.onException(SingleConnectionFactory.java:670)

• waiting to lock <1ea19285> (a java.lang.Object) owned by "clientResponseListener-16" t@242
  at org.springframework.jms.connection.SingleConnectionFactory$AggregatedExceptionListener.onException(SingleConnectionFactory.java:671)
• locked <1768ffa4> (a java.lang.Object)
  at com.tibco.tibjms.TibjmsConnection._invokeOnExceptionCallback(TibjmsConnection.java:2054)
  at com.tibco.tibjms.TibjmsConnection._onDisconnected(TibjmsConnection.java:2394)
  at com.tibco.tibjms.TibjmsConnection$ServerLinkEventHandler.onEventDisconnected(TibjmsConnection.java:349)
  at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:330)
  at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)

Locked ownable synchronizers:

• None

---

Affects: 4.2.7

Issue Links:

• Memory leak with SimpleMessageListenerContainer and ChainedExceptionListener on Single/CachingConnectionFactory [SPR-10397] #15030 Memory leak with SimpleMessageListenerContainer and ChainedExceptionListener on Single/CachingConnectionFactory
• ConcurrentModificationException in SingleConnectionFactory$AggregatedExceptionListener [SPR-13421] #18000 ConcurrentModificationException in SingleConnectionFactory$AggregatedExceptionListener

Referenced from: commits 6d55b3a, a9a4d7c

[spring-projects-issues]

Anup Tagalpallewar commented

Hi Juergen,

Can you please suggest any workaround that might help avoid the deadlock?

Thanks

[spring-projects-issues]

Juergen Hoeller commented

I've addressed this through moving AggregatedExceptionListener's onException delegate calls outside of the connectionMonitor there, just taking a temporary copy of the current listeners within the monitor now. Fixed for 5.0 RC3, and to be backported to 4.3.10.

I don't think Tibco should be using such a hard lock for onException callbacks in the first place... but I doubt they'll change that any time soon. So not taking a lock for those callbacks on Spring's side is the only thing we can do there.

As a workaround, you could try to get rid of your SingleConnectionFactory setup. DefaultMessageListenerContainer manages its own shared connection in any case, so you might not actually benefit from the additional SingleConnectionFactory...