Extend service to suggest addons via generic IP scan

[holgerfriedrich]

PR #3920 introduced a (partly) generic addon finder, which allows to discover devices by sending single IP frames. This is useful for devices which do not actively announce and require a search request to a multicast address. Maybe the word scanning is a bit misleading, at the moment it is just a single search frame, and only once at start-up in the current implementation.

Note: if case you can use mdns or upnp, there are separate addon finders for this! If you want to detect a service running locally, there is one for processes as well.

The implementation is rather tuned to a single use right now - detecting KNX installations - but is intended to be extended towards release 4.2. This issue is to collect requirements from other addons which can be used when starting a generalization.

The configuration is via addon.xml and looks like this (see #3948 for developer documentation):

	<discovery-method>
		<discovery-method>
			<service-type>ip</service-type>
			<discovery-parameters>
				<discovery-parameter>
					<name>type</name>
					<value>ipMulticast</value>
				</discovery-parameter>
				<discovery-parameter>
					<name>destIp</name>
					<value>224.0.23.12</value>
				</discovery-parameter>
				<discovery-parameter>
					<name>destPort</name>
					<value>3671</value>
				</discovery-parameter>
				<discovery-parameter>
					<name>request</name>
					<value>0x06 0x10 0x02 0x01 0x00 0x0e 0x08 0x01 $srcIp $srcPort</value>
				</discovery-parameter>
				<discovery-parameter>
					<name>timeoutMs</name>
					<value>5000</value>
				</discovery-parameter>
			</discovery-parameters>
			<match-properties>
				<match-property>
					<name>response</name>
					<regex>.*</regex>
				</match-property>
			</match-properties>
		</discovery-method>
	</discovery-methods>

Frames to be sent are specified in the parameter section, and dynamic replacement of source IP and port is possible. Any return frame is matched, only the catch all .* is supported.

addon	detection method	scan interval	dynamic data	parsed response	comment
knx	mulitcast frame (hex data)	once	srcIp, srcPort	.*	#3912 (merged)
ipcamera	multicast frame (xml)	tbd	randomNumber	.*	#3943 openhab/openhab-addons#16090 (merged)
govee	multicast frame	tbd	no	.*	requires listening on port 4002, openhab/openhab-addons#16109 (merged)
zway	unicast, scan local network	tbd	none	http code 200	see below
danfoss	broadcast	tbd	none	static content	#4036 (merged)

(table to be continued)

Features to be developed (incomplete list):

• plaintext request instead of hex bytes ✅
• send at regular intervals (add-on specific setting)
• real regex matching for result frames (match packet as hex stream, and loop over all result frames until match is found or timeout occurs)
• new dynamic data, e.g. randomNumber of certain length (✅ for uuid sting)
• possibly caching to avoid sending the same query twice
• skip detection if binding is already installed ✅
• unicast
• broadcast ✅
• parallel sending (avoid that suggestion is missed if the checker was not running before wizard starts - UI could also be improved to wait until all checks finished) [WIP] IP Add-on Finder: make scans asynchronously #4094
• respecting network config from Add network-interface context #3981
• cleaning up the log levels (debug for status messages, trace for data on the network)

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/how-to-get-an-addon-to-show-up-in-the-new-addon-suggestion-finder-list/152127/13

[Skinah]

To summarize what is needed for the ipcamera addon as discussed in the forum link:

To discover ONVIF cameras it uses port 3702 and IP multicast address 239.255.255.250 using what is called WS-Discovery. It is just the following SOAP/XML packet sent.

<?xml version="1.0" encoding="UTF-8"?><e:Envelope xmlns:e="http://www.w3.org/2003/05/soap-envelope" xmlns:w="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:d="http://schemas.xmlsoap.org/ws/2005/04/discovery" xmlns:dn="http://www.onvif.org/ver10/network/wsdl"><e:Header><w:MessageID>uuid:randomUUID()</w:MessageID><w:To e:mustUnderstand="true">urn:schemas-xmlsoap-org:ws:2005:04:discovery</w:To><w:Action a:mustUnderstand="true">http://schemas.xmlsoap.org/ws/2005/04/discovery/Probe</w:Action></e:Header><e:Body><d:Probe><d:Types xmlns:d="http://schemas.xmlsoap.org/ws/2005/04/discovery" xmlns:dp0="http://www.onvif.org/ver10/network/wsdl">dp0:NetworkVideoTransmitter</d:Types></d:Probe></e:Body></e:Envelope>

Challenges identified are:

1. In the above XML packet you need to use the function java.util.UUID.randomUUID() to replace the text randomUUID() It needs to be different for each scan.
2. How to enter in XML contents into the addon.xml file without causing issues.
3. What about when a reply could match multiple bindings, do we do the scan multiple times and then use a different filter, or can they be combined? Doorbird is a binding for that brand of ipcamera which can also do ONVIF, so it would match two bindings. Do we scan once and then filter the cached results differently in each binding, or do the whole scan multiple times?

Good news is that because the XML states NetworkVideoTransmitter in the request, all replies will only be from devices that the binding can talk to. No filtering needed on replies since this is just for just putting the binding forward to be installed where further logic can be used.

[mherwege]

3. What about when a reply could match multiple bindings, do we do the scan multiple times and then use a different filter, or can they be combined? Doorbird is a binding for that brand of ipcamera which can also do ONVIF, so it would match two bindings. Do we scan once and then filter the cached results differently in each binding, or do the whole scan multiple times?

If both bindings define the same discovery criteria, they will both be suggested. It is for the user to decide what to install. No 2 scans are required for this.

[holgerfriedrich]

@Skinah Would the following string work? (this is the result of a draft implementation)
Is UUID fine?

<?xml version="1.0" encoding="UTF-8"?><e:Envelope xmlns:e="http://www.w3.org/2003/05/soap-envelope" xmlns:w="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:d="http://schemas.xmlsoap.org/ws/2005/04/discovery" xmlns:dn="http://www.onvif.org/ver10/network/wsdl"><e:Header><w:MessageID>uuid:1534bd9d-f9c1-45cc-a747-311d6a76db3b<?xml version="1.0" encoding="UTF-8"?><e:Envelope xmlns:e="http://www.w3.org/2003/05/soap-envelope" xmlns:w="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:d="http://schemas.xmlsoap.org/ws/2005/04/discovery" xmlns:dn="http://www.onvif.org/ver10/network/wsdl"><e:Header><w:MessageID>uuid:)</w:MessageID><w:To e:mustUnderstand="true">urn:schemas-xmlsoap-org:ws:2005:04:discovery</w:To><w:Action a:mustUnderstand="true">http://schemas.xmlsoap.org/ws/2005/04/discovery/Probe</w:Action></e:Header><e:Body><d:Probe><d:Types xmlns:d="http://schemas.xmlsoap.org/ws/2005/04/discovery" xmlns:dp0="http://www.onvif.org/ver10/network/wsdl">dp0:NetworkVideoTransmitter</d:Types></d:Probe></e:Body></e:Envelope>

[holgerfriedrich]

@stefan-hoehn Mechanism for Govee
#3920 (comment)
can be discussed here.

Developer docs are not yet merged, see #3948.

request does only support hex bytes by now, so you could make it work if you use
<value>0x7B 0x22 0x6D 0x73 0x67 0x22 0x3A 0x20 0x7B 0x22 0x63 0x6D 0x64 0x22 0x3A 0x20 0x22 0x73 0x63 0x61 0x6E 0x22 0x2C 0x20 0x22 0x64 0x61 0x74 0x61 0x22 0x3A 0x20 0x7B 0x22 0x61 0x63 0x63 0x6F 0x75 0x6E 0x74 0x5F 0x74 0x6F 0x70 0x69 0x63 0x22 0x3A 0x20 0x22 0x72 0x65 0x73 0x65 0x72 0x76 0x65 0x22 0x7D 0x7D 0x7D</value>

Not sure about port 4002.... do we have to listen on 4002 or is if source port of the response?

[holgerfriedrich]

@stefan-hoehn just updated the comment above to include an example....

[stefan-hoehn]

This is how it works:

• send a multicast request to port 4001
• and the clients send and asynchronous response back on port 4002, so we need to listen explicitly on 4002
• the response then is as follows

{
   "msg":{
      "cmd":"scan",
      "data":{
        "ip":"192.168.1.23",
        "device":"1F:80:C5:32:32:36:72:4E",
        "sku":"Hxxxx",
        "bleVersionHard":"3.01.01",
        "bleVersionSoft":"1.03.01",
        "wifiVersionHard":"1.00.10",
        "wifiVersionSoft":"1.02.03"
      }
    }
  }
 

But I think noticing that we get ANY response would be enough, I would say.

[mherwege]

@holgerfriedrich Looking at this, and the ipcamera request, I think an alternative (non-hex) request should be supported.

[holgerfriedrich]

@holgerfriedrich Looking at this, and the ipcamera request, I think an alternative (non-hex) request should be supported.

:-)

Looks Like.... But we need to make sure to escape tags - otherwise the XML parser might do something wrong....

[mherwege]

@mherwege I don’t think that is the responsibility of the code. When defining the discovery criteria in the xml, it is sufficient to properly escape the special characters: https://stackoverflow.com/questions/1091945/what-characters-do-i-need-to-escape-in-xml-documents
But that would be true for any of the finders.

[holgerfriedrich]

@mherwege Yes, lets go for a requestPlain which uses org.openhab.core.util.StringUtils.StringUtilsunEscapeXml()..
Developers of bindings need to escape the characters mentioned here by hand.
Better than using the hex editor to transform the query into something unreadable.

[stefan-hoehn]

I can confirm that the implementation works well already for the Govee-Binding. See #16109

[holgerfriedrich]

andrewfg commented Dec 27, 2023

@holgerfriedrich the ZWay binding is probably a candidate to be suggested via your IP addon finder. => WDYT?

https://github.com/openhab/openhab-addons/tree/main/bundles/org.openhab.binding.zway#discovery

This is about the zway gateways connected to the LAN.

The documentation of the zway binding states the following:

Z-Way doesn't support any discovery protocol like UPnP for this purpose. That is why first all IP addresses in local network are checked on port 8083. If the server answers, a ZAutomation request (/ZAutomation/api/v1/status) is performed to ensure, the found server runs Z-Way.

This would be the first real "IP scan", i.e. iterating over all local IP addresses finding a gateway.

It would require a few new features:

• new type "unicast"
• allowing a dynamic IP range (e.g. local subnet of each interface) - but still limiting the number of scanned hosts to avoid the problems we have seen with Docker
• parsing the response (likely we need to match a http status code)
• and because this is creating some load, I would also suggest that we do this only if the binding is not installed

Did I miss something, @andrewfg ?

[andrewfg]

Did I miss something

I think it pings all port 8083 on the subnet, and if the port is open it does a specific HTTP GET for a URL on that host:8083, and if it returns HTTP 200 it would become a suggestion.

[holgerfriedrich]

I think it pings all port 8083 on the subnet, and if the port is open it does a specific HTTP GET for a URL on that host:8083, and if it returns HTTP 200 it would become a suggestion.

Yes, this sounds reasonable, thanks.

A subnet could be quite large, I would limit it a /24 and exclude the broadcast address (such that we have to scan 255 IPs per interface at max).

[mherwege]

I don’t think we should allow scans on all interface. See discussion here, how it can cause drama with Docker: https://community.openhab.org/t/unsuccess-openhab-updated-from-4-0-2-to-4-1-0-on-docker-openhab-unresponsive/152282/24

OH does have a setting for the primary IP. Could we restrict it to thecsubnet of that iP?

I even think we may want to limit the current IP finder implementation to only use one interface (or make that configurable).

[mlobstein]

If regex matching is implemented then the epsonprojector and sonyprojector addons could be detected as many newer models respond to Control4 SDDP discovery probes as described here:

https://github.com/sammck/sddp-discovery-protocol/blob/main/README.md

SDDP is similar to UPNP/SSDP in that a multicast search frame is sent and any supported devices respond back to the multicast address with a packet containing device information. By using a regex these packets would be filtered to determine the device type and manufacturer.

[andrewfg]

@mlobstein I created #4234 for this..