Jetty Buffer Exception / Homematic Discovery

[justsendit]

I am trying to connect openhab to my homematic setup (>100 HmIP Devices).
When i started with less devices it worked just fine, but now i cannot add any(thing).

In the openhab events.log it tells me the device serial is not found on the gateway:
2020-02-05 16:20:03.830 [WARN ] [ernal.handler.HomematicBridgeHandler] - Device with address '[SERIAL]' not found on gateway 'CCU3'

Restarting the openhab2 service it throws the following error, which i think is the issue:

2020-02-05 16:20:03.810 [ERROR] [very.HomematicDeviceDiscoveryService] - java.lang.IllegalArgumentException: Buffering capacity 2097152 exceeded
java.io.IOException: java.lang.IllegalArgumentException: Buffering capacity 2097152 exceeded
at org.openhab.binding.homematic.internal.communicator.client.XmlRpcClient.sendMessage(XmlRpcClient.java:98) ~[?:?]
at org.openhab.binding.homematic.internal.communicator.client.XmlRpcClient.sendMessage(XmlRpcClient.java:102) ~[?:?]
at org.openhab.binding.homematic.internal.communicator.client.XmlRpcClient.sendMessage(XmlRpcClient.java:67) ~[?:?]
at org.openhab.binding.homematic.internal.communicator.client.RpcClient.listDevices(RpcClient.java:168) ~[?:?]
at org.openhab.binding.homematic.internal.communicator.AbstractHomematicGateway.getDeviceDescriptions(AbstractHomematicGateway.java:448) ~[?:?]
at org.openhab.binding.homematic.internal.communicator.AbstractHomematicGateway.loadAllDeviceMetadata(AbstractHomematicGateway.java:372) ~[?:?]
at org.openhab.binding.homematic.internal.discovery.HomematicDeviceDiscoveryService.lambda$0(HomematicDeviceDiscoveryService.java:210) ~[?:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_232]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_232]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_232]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Buffering capacity 2097152 exceeded
at org.eclipse.jetty.client.util.FutureResponseListener.getResult(FutureResponseListener.java:118) ~[?:?]
at org.eclipse.jetty.client.util.FutureResponseListener.get(FutureResponseListener.java:101) ~[?:?]
at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:685) ~[?:?]
at org.openhab.binding.homematic.internal.communicator.client.XmlRpcClient.sendMessage(XmlRpcClient.java:84) ~[?:?]
... 13 more
Caused by: java.lang.IllegalArgumentException: Buffering capacity 2097152 exceeded
at org.eclipse.jetty.client.util.BufferingResponseListener.onHeaders(BufferingResponseListener.java:83) ~[?:?]
at org.eclipse.jetty.client.ResponseNotifier.notifyHeaders(ResponseNotifier.java:98) ~[?:?]
at org.eclipse.jetty.client.ResponseNotifier.notifyHeaders(ResponseNotifier.java:90) ~[?:?]
at org.eclipse.jetty.client.HttpReceiver.responseHeaders(HttpReceiver.java:269) ~[?:?]
at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.headerComplete(HttpReceiverOverHTTP.java:256) ~[?:?]
at org.eclipse.jetty.http.HttpParser.parseFields(HttpParser.java:1192) ~[?:?]
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1475) ~[?:?]
at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.parse(HttpReceiverOverHTTP.java:172) ~[?:?]
at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:135) ~[?:?]
at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:73) ~[?:?]
at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:133) ~[?:?]
at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:154) ~[?:?]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ~[?:?]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) ~[?:?]
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) ~[?:?]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) ~[?:?]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) ~[?:?]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) ~[?:?]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) ~[?:?]
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:367) ~[?:?]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782) ~[?:?]
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918) ~[?:?]
... 1 more

I've read there is i fixed buffer limit of 2MB within java which might cause this. I believe jetty is invoked by the discovery service to query the CCU3 for its devices but the response is too big to fit the buffer, so it just crashes.

If i only add the CCU to my things file and nothing else, it works just fine.

Is there anything that can be done to this? I found some hints for jetty on setting a larger buffer size, but i don't know how this can be done with openhab.

Best regards
Andreas

[MHerbst]

Is there anything that can be done to this? I found some hints for jetty on setting a larger buffer size, but i don't know how this can be done with openhab.

As far as I can see the method that reads the responses from the CCU needs some modification to allows the processing of messages that are larger than 2 GB.

[joerghohn]

As statet by the OP, this seems not to be homematic specific as I have the same buffer error with the Unifi Addon:

2020-02-15 15:16:05.797 [hingStatusInfoChangedEvent] - 'unifi:controller:7ef1983d' changed from UNINITIALIZED (DISABLED) to INITIALIZING
2020-02-15 15:16:06.687 [hingStatusInfoChangedEvent] - 'unifi:controller:7ef1983d' changed from INITIALIZING to ONLINE
2020-02-15 15:16:08.611 [WARN ] [.handler.UniFiControllerThingHandler] - Unhandled exception while refreshing the UniFi Controller unifi:controller:7ef1983d - java.lang.IllegalArgumentException: Buffering capacity 2097152 exceeded
2020-02-15 15:16:08.618 [hingStatusInfoChangedEvent] - 'unifi:controller:7ef1983d' changed from ONLINE to OFFLINE (COMMUNICATION_ERROR): java.lang.IllegalArgumentException: Buffering capacity 2097152 exceeded
2020-02-15 15:16:20.674 [WARN ] [.handler.UniFiControllerThingHandler] - Unhandled exception while refreshing the UniFi Controller unifi:controller:7ef1983d - java.lang.IllegalArgumentException: Buffering capacity 2097152 exceeded

[9037568]

This solution appears to be what's needed.

[MHerbst]

@9037568 I have found this solution on Stack Overflow, too. Should probably work with Homematic.

@joerghohn At least the Homematic binding uses its own routine for the communication via Jetty. I would assume that the Unify binding does the same. Probably it would better to create a separate issue for the Unify binding.

[MHerbst]

@9037568 I have modified the binding and you can download a test version from here: https://github.com/MHerbst/openhab-addons-test/raw/master/org.openhab.binding.homematic-2.5.3-SNAPSHOT.jar
I would like to get some feedback before I create a PR because I can't test it.

[MHerbst]

@J-N-K @cpmeister My PR did not automatically close this issue. Could one of you do it, since I lack the necessary authority? Thanks.

[J-N-K]

Probably because "Solves" is not a keyword for GitHub. "Fixes" and "Closes" work.