Eclipse Smarthome Implementation #85

ThomDietrich · 2018-04-25T17:03:43Z

@ThomDietrich I expect some homie implementations to skip certain topics ($fw,$implementation,etc) and I'm also expecting the spec to grow some more topics. That's not an issue at all for the implementation that I'm realizing right now.

It will break, if you change topic hierarchies or key topics. And that's what I mean.

Btw, I'm struggling with node instances (arrays) and may leave them out for now. It is not obvious for me, if node attributes are repeated for each instance. And if it is an array, what consists the not indexed node of. Is it only the attributes or can it have a value as well?
Example:
homie/mydevice/node
homie/mydevice/node/$name="name"
homie/mydevice/node_0/$name="name0"
Is this a valid thing: home/mydevice/node and home/mydevice/node/set?

The text was updated successfully, but these errors were encountered:

ThomDietrich · 2018-04-25T17:07:53Z

The first part sounds correct to me! No problem after all. Of course this is how the development of homie is anticipated (see #46).

Regarding arrays I want to mention @marvinroger as I did never touch them and I know that he had some recent thought about them.

davidgraeff · 2018-04-26T15:11:14Z

It is official now: eclipse-archived/smarthome#5450
Some more work needs to be done though.

davidgraeff · 2018-04-30T15:02:53Z

FYI: The implementation (except node instances/arrays, which I do not plan to do atm) is done. I have not yet performed any real tests, because I do not have a MQTT setup yet. If anybody else want to test the next days, please do so and report back in the linked eclipse smarthome issue.

Cheers,
David

davidgraeff · 2018-05-02T20:34:42Z

:(
https://github.com/andsel/moquette/issues/401
The eclipse smarthome embedded MQTT broker cannot process topics starting with a "$".

ThomDietrich · 2018-05-02T22:52:21Z

Wait... what?
The issue wasn't informative. Do we have a problem? 😅 Any chance to solve it?

davidgraeff · 2018-05-02T22:58:01Z

The problem: It is not possible to subscribe to topics containing a "$".

It is not the moquette broker, as I anticipated originally. It is worse. It's the antique paho Mqtt client library version that is used in ESH/Openhab. Unfortunately there is no new release since half a year, the last commit is even 9 months ago. The developer is apparently concentrating on the mqttv5 part of the library (which is incompatible, API and protocol wise). ESH is not homie convention compatible at the moment.

bodiroga · 2018-05-03T09:06:00Z

Hi @davidgraeff!

What Paho library version is used in ESH MQTT transport bundle?

1-2 years ago @gorootde started with the development of a Homie binding for openHAB, using the 1.1.0 version of Paho, and it works fine. Here you have a link to his repository: https://github.com/gorootde/openhab2-addons/tree/bindings/homie/addons/binding/org.openhab.binding.homie A have made some improvements to the binding for my personal use (not released yet, sorry, although I would like to extend your work in the Homie 3.X binding to support Homie 2.0.1 devices) and I didn't found any error subscribing to topics containing a "$" 😕

Is the library version being used older that 1.1?

Many thanks for your work!

Aitor

davidgraeff · 2018-05-03T09:28:42Z

It is the latest available release 1.2. If someone has the time to test the raw paho library that would be awesome. In ESH I'm using the Moquette server, the paho library and on top an abstraction layer. Any of those components or the composition could be the problem.

bodiroga · 2018-05-03T11:36:30Z

Hi again @davidgraeff!

I will try to make some test again using the 1.2.0 version of the library, but I'm pretty sure that it doesn't have any problem subscribing to topics with "$" 😞 What tests should I do?

In the meantime, have you tried using one of the publicly available MQTT brokers? Eclipse has one at "tcp://iot.eclipse.org" (1883 or 8883) and HiveMQ also offers a public broker at "tcp://broker.hivemq.com:1883".

euphi · 2018-05-03T15:03:30Z

I haven't found any restriction in the paho sources (https://github.com/eclipse/paho.mqtt.java/blob/master/org.eclipse.paho.client.mqttv3/src/main/java/org/eclipse/paho/client/mqttv3/MqttTopic.java and some other files like MqttAsyncClient.java)?

@davidgraeff are you sure your problem is on client side? Which server are you using?

bodiroga · 2018-05-03T15:26:53Z

Hi!

I've written a very simple client that subscribes to the topic "homie/homie_test/$online" in the "iot.eclipse.org" broker and everything seems to work fine. This is the 'stupid' code:

import java.util.UUID;

import org.eclipse.paho.client.mqttv3.IMqttDeliveryToken;
import org.eclipse.paho.client.mqttv3.MqttCallbackExtended;
import org.eclipse.paho.client.mqttv3.MqttClient;
import org.eclipse.paho.client.mqttv3.MqttConnectOptions;
import org.eclipse.paho.client.mqttv3.MqttException;
import org.eclipse.paho.client.mqttv3.MqttMessage;
import org.eclipse.paho.client.mqttv3.MqttSecurityException;

public class MqttTest implements MqttCallbackExtended {
	
	MqttClient client;
		
	public MqttTest(String brokerURI) {
		System.out.println("Trying to stablish a connection to: " + brokerURI);
				
	    String sessionQualifier = String.format("%s", UUID.randomUUID());
	    MqttConnectOptions opts = new MqttConnectOptions();
	                
	    try {
	    	client = new MqttClient(brokerURI, sessionQualifier);
		    client.setCallback(this);
			client.connect(opts);
		} catch (MqttSecurityException e) {
			System.out.println("Client error: " + e.getMessage());
			e.printStackTrace();
		} catch (MqttException e) {
			System.out.println("Client error: " + e.getMessage());
		}
	}

	@Override
	public void messageArrived(String topic, MqttMessage message) {
		System.out.println(String.format("Message arrived to topic '%s' with message '%s'", topic, message.toString()));
	}

	@Override
	public void connectionLost(Throwable arg0) {
		System.out.println("Connection with the broker lost");
			
	}

	@Override
	public void deliveryComplete(IMqttDeliveryToken arg0) {		
	}

	@Override
	public void connectComplete(boolean arg0, String arg1) {
		System.out.println("Connection with the broker stablished");
	    String faultyTopic = "homie/homie_test/$online";
		System.out.println("Subscribing to topic: " + faultyTopic);
		try {
			client.subscribe(faultyTopic);
		} catch (MqttException e) {
			e.printStackTrace();
		}
	}
	

	public static void main(String[] args) {
		
		String brokerHost = "iot.eclipse.org";
		int brokerPort = 1883;
		String brokerURI = String.format("tcp://%s:%s", brokerHost, brokerPort);
		
		new MqttTest(brokerURI);
		
	}
}

And here you have the execution output:

Trying to stablish a connection to: tcp://iot.eclipse.org:1883
Connection with the broker stablished
Subscribing to topic: homie/homie_test/$online
Message arrived to topic 'homie/homie_test/$online' with message 'false'

Nothing strange, so the problem must be somewhere else 🤔

davidgraeff · 2018-05-03T16:01:47Z

I had the suspicion that Moquette is the culprit, but the command line mosquito client was able to publish and subscribe "$" topics. Example code of my test suite:

embeddedConnection.publish(deviceID + "/$homie", "3.0".getBytes())
embeddedConnection.publish(deviceID + "/homieÄ:?ß", "3.0".getBytes())

I'm publishing the default $homie topic as well as a more or less random utf8 topic.
Later on I test three different things:

connection.subscribe(deviceID + "/#",
                (topic, payload) -> succeed_the_test()).get(200, TimeUnit.MILLISECONDS);
connection.subscribe(deviceID + "/$homie",
                (topic, payload) -> succeed_the_test()).get(200, TimeUnit.MILLISECONDS);
connection.subscribe(deviceID + "/homieÄ:?ß",
                (topic, payload) -> succeed_the_test()).get(200, TimeUnit.MILLISECONDS);

Only the second test times out. Paho is not able to receive anything from Moquette for any "$" topic.

bodiroga · 2018-05-03T18:44:05Z

Ummmm... I'm trying a local Moquette configuration and it's driving me nuts 😢

I have followed this steps (https://github.com/andsel/moquette#1-minute-set-up):

Download the .tar file (https://bintray.com/artifact/download/andsel/generic/distribution-0.10-bundle-tar.tar.gz)
Unzip the file
Start the .sh script executing "./moquette.sh"

Then, I have published a retained message using the command line mosquitto client:

mosquitto_pub -h localhost -t "test/test" -r -m "ok!"

And I have attempted to read the message back with the mosquitto_sub command:

mosquitto_sub -h localhost -v -t "test/test"

But I'm not able to receive the retained message! This is what the Mosquitto terminal tells me:

1156562 [nioEventLoopGroup-3-1] INFO  ProtocolProcessor  - Processing CONNECT message. CId=mosqsub/7823-aitor-port, username=null
1156562 [nioEventLoopGroup-3-1] INFO  ProtocolProcessor  - Configuring connection. CId=mosqsub/7823-aitor-port
1156562 [nioEventLoopGroup-3-1] INFO  ProtocolProcessor  - Sending connect ACK. CId=mosqsub/7823-aitor-port
1156562 [nioEventLoopGroup-3-1] ERROR MemorySessionStore  - Can't find the session for client <mosqsub/7823-aitor-port>
1156562 [nioEventLoopGroup-3-1] INFO  ProtocolProcessor  - The connect ACK has been sent. CId=mosqsub/7823-aitor-port
1156563 [nioEventLoopGroup-3-1] ERROR MemorySessionStore  - Can't find the session for client <mosqsub/7823-aitor-port>
1156563 [nioEventLoopGroup-3-1] INFO  ProtocolProcessor  - Cleaning session. CId=mosqsub/7823-aitor-port
1156563 [nioEventLoopGroup-3-1] ERROR MemorySessionStore  - Fooooooooo <mosqsub/7823-aitor-port>
1156563 [nioEventLoopGroup-3-1] INFO  MemorySessionStore  - Removing stored messages with QoS 1 and 2. ClientId=mosqsub/7823-aitor-port
1156563 [nioEventLoopGroup-3-1] INFO  MemorySessionStore  - Wiping existing subscriptions. ClientId=mosqsub/7823-aitor-port
1156563 [nioEventLoopGroup-3-1] INFO  ProtocolProcessor  - The CONNECT message has been processed. CId=mosqsub/7823-aitor-port, username=null
1156563 [nioEventLoopGroup-3-1] INFO  ProtocolProcessor  - Processing SUBSCRIBE message. CId=mosqsub/7823-aitor-port, messageId=1
1156563 [nioEventLoopGroup-3-1] INFO  ProtocolProcessor  - Client will be subscribed to the topic CId=mosqsub/7823-aitor-port, username=null, messageId=1, topic=test/test
1156563 [nioEventLoopGroup-3-1] INFO  ProtocolProcessor  - Creating and storing subscriptions CId=mosqsub/7823-aitor-port, messageId=1, topics=[MqttTopicSubscription[topicFilter=test/test, qualityOfService=AT_MOST_ONCE]]
1156563 [nioEventLoopGroup-3-1] INFO  ClientSession  - Adding new subscription. ClientId=mosqsub/7823-aitor-port, topics=test/test, qos=AT_MOST_ONCE
1156563 [nioEventLoopGroup-3-1] INFO  ProtocolProcessor  - Sending SUBACK response CId=mosqsub/7823-aitor-port, messageId=1
1156564 [nioEventLoopGroup-3-1] INFO  ProtocolProcessor  - Retrieving retained messages CId=mosqsub/7823-aitor-port, topics=test/test

If I publish a new message while the mosquitto_sub client is connected, everything works fine, but retained messages are not delivered. However, the following commands work perfectly:

mosquitto_pub -h iot.eclipse.org -t "test/test" -r -m "ok"
mosquitto_sub -h iot.eclipse.org -v -t "test/test"

Can anyone verify this simple test? Do you confirm my results? @davidgraeff, what are your results using the "iot.eclipse.org" broker instead of the ESH embedded broker?

PS: Perhaps we are getting too off-topic...

davidgraeff · 2018-05-03T18:57:54Z

Moquette only retains messages with QoS != 0. I've overlooked that and it took me some time. That's actually spec conform, the issue was closed on the moquette repository.

One other note: I'm using paho in asynchronous mode. Might be a bug in that implementation.

bodiroga · 2018-05-03T19:54:20Z

Wow, I didn't know that, many thanks for the hint David!

Just for the sake of completeness, here you have a link where it was discussed in the Moquette repository: https://github.com/andsel/moquette/issues/125. It's strange because they seem to be the only ones doing it, although they fully conform to the specification.

Anyway, now I'm not able to confirm your results, sorry. My test have been done using:

Moquette broker. Downloaded and executed using the steps provided in the GitHub repository.
Mosquitto command line tool for publishing the retained message.
mosquitto_pub -h localhost -t "homie/homie_test/\$homie" -m "3.0.0" -r -q 1
Simple client build using the Java paho library (Code here: Eclipse Smarthome Implementation #85 (comment)). It subscribes to the "homie/homie_test/$homie" topic.

The small client is able to receive all messages published to the topic containing the "$", I can't see anything wrong 🤔

Trying to stablish a connection to: tcp://localhost:1883
Connection with the broker stablished
Subscribing to topic: homie/homie_test/$homie
Message arrived to topic 'homie/homie_test/$homie' with message '3.0.0'
Message arrived to topic 'homie/homie_test/$homie' with message '2.0.1'
Message arrived to topic 'homie/homie_test/$homie' with message '3.0.0'

Do you want me to test anything in particular? 👍

davidgraeff · 2018-05-03T20:29:20Z

Thanks a lot for investigating. Could you try the paho asynchronous API as well? Especially moquette and paho. That's the last piece.

If it's not in there, it's related to the esh abstraction layer. What is also done in esh is: paho is configured to use persistence. Maybe the persistence layer has an issue with dollar sign topics. I'll investigate.

bodiroga · 2018-05-04T08:30:29Z

Here I am again!

New tests done with the Paho Async API, here you can see the client implementation (just in case someone wants to try it out):

import java.util.UUID;

import org.eclipse.paho.client.mqttv3.IMqttActionListener;
import org.eclipse.paho.client.mqttv3.IMqttDeliveryToken;
import org.eclipse.paho.client.mqttv3.IMqttToken;
import org.eclipse.paho.client.mqttv3.MqttAsyncClient;
import org.eclipse.paho.client.mqttv3.MqttCallback;
import org.eclipse.paho.client.mqttv3.MqttConnectOptions;
import org.eclipse.paho.client.mqttv3.MqttException;
import org.eclipse.paho.client.mqttv3.MqttMessage;
import org.eclipse.paho.client.mqttv3.MqttSecurityException;

public class MqttTest {
	
	MqttAsyncClient client;
		
	public MqttTest(String brokerURI) {
		System.out.println("Trying to stablish a connection to: " + brokerURI);
				
		String sessionQualifier = String.format("%s", UUID.randomUUID());
		MqttConnectOptions opts = new MqttConnectOptions();
		opts.setCleanSession(true);
		
		try {
			client = new MqttAsyncClient(brokerURI, sessionQualifier);
	    	
	    	client.setCallback(new MqttCallback() {
	    		
				@Override
				public void connectionLost(Throwable arg0) {
					System.out.println("Connection to the broker lost");
					System.exit(0);
				}

				@Override
				public void deliveryComplete(IMqttDeliveryToken arg0) {	
				}

				@Override
				public void messageArrived(String topic, MqttMessage message) throws Exception {
					System.out.println(String.format("Message arrived to topic '%s' with message '%s'", topic, message.toString()));
				}

			});
	    	
			client.connect(opts, null, new IMqttActionListener() {
				public void onSuccess(IMqttToken asyncActionToken) {
					System.out.println("Connection with the broker stablished");
					String faultyTopic = "homie/homie_test/$homie";
					System.out.println("Subscribing to topic: " + faultyTopic);
					
			    	try {
			    		client.subscribe(faultyTopic, 1);
			    	} catch (MqttException e) {
			    		e.printStackTrace();
					}
				}

				public void onFailure(IMqttToken asyncActionToken, Throwable exception) {
					System.out.println("Client couldn't connecto to the broker");
					System.exit(0);
				}
			});
			
		} catch (MqttSecurityException e) {
			System.out.println("Client error: " + e.getMessage());
		} catch (MqttException e) {
			System.out.println("Client error: " + e.getMessage());
		}
	}

	public static void main(String[] args) {
		
		String brokerHost = "localhost";
		int brokerPort = 1883;
		String brokerURI = String.format("tcp://%s:%s", brokerHost, brokerPort);
		
		new MqttTest(brokerURI);	
	}
}

And the output:

Trying to stablish a connection to: tcp://localhost:1883
Connection with the broker stablished
Subscribing to topic: homie/homie_test/$homie
Message arrived to topic 'homie/homie_test/$homie' with message '3.0.0'

Nothing wrong, nothing suspicious, works fine.

davidgraeff · 2018-05-04T10:52:13Z

Thanks a lot. I have done my homework as well and analysed the abstraction layer, by adding a ton more test cases. And indeed the problem is that simple line:
if (topic.matches(consumerList.regexMatchTopic))
with topic=homie/device123/$name and regexMatchTopic=homie/device123/$name.

The dollar sign makes the regex expect the end of the string which results in no match.

bodiroga · 2018-05-04T11:20:26Z

So the ESH MQTT transport layer needs to be fixed then? Do you know how to solve it? Let's hope that the future PR will be accepted as soon as possible 👍

Thanks for your work David!

davidgraeff · 2018-05-04T12:03:22Z

It is solved already with a PR that I will soon open. I need to finish work first.

davidgraeff · 2018-05-05T12:35:47Z

The MQTT with homie ESH implementation is done on my side now and is provided in 4 different PRs. Unfortunately I needed to redesign API of an existing class (API break). That may result in a longer discussion which can be followed here: eclipse-archived/smarthome#5535

davidgraeff · 2018-10-24T10:18:00Z

Done. Can be closed.

marvinroger · 2018-10-24T13:30:13Z

That's awesome. Thanks @davidgraeff and everyone involved. 👍

ThomDietrich mentioned this issue Apr 25, 2018

Some thoughts beyond v3.0 #62

Closed

ThomDietrich mentioned this issue Apr 25, 2018

Openhab2 binding - Anyone already working on one? #10

Closed

ThomDietrich added the implementation label May 3, 2018

marvinroger closed this as completed Oct 24, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Eclipse Smarthome Implementation #85

Eclipse Smarthome Implementation #85

ThomDietrich commented Apr 25, 2018

ThomDietrich commented Apr 25, 2018 •

edited

Loading

davidgraeff commented Apr 26, 2018

davidgraeff commented Apr 30, 2018

davidgraeff commented May 2, 2018

ThomDietrich commented May 2, 2018

davidgraeff commented May 2, 2018 •

edited

Loading

bodiroga commented May 3, 2018

davidgraeff commented May 3, 2018

bodiroga commented May 3, 2018 •

edited

Loading

euphi commented May 3, 2018

bodiroga commented May 3, 2018

davidgraeff commented May 3, 2018 •

edited

Loading

bodiroga commented May 3, 2018

davidgraeff commented May 3, 2018 •

edited

Loading

bodiroga commented May 3, 2018

davidgraeff commented May 3, 2018

bodiroga commented May 4, 2018

davidgraeff commented May 4, 2018 •

edited

Loading

bodiroga commented May 4, 2018

davidgraeff commented May 4, 2018

davidgraeff commented May 5, 2018

davidgraeff commented Oct 24, 2018

marvinroger commented Oct 24, 2018

Eclipse Smarthome Implementation #85

Eclipse Smarthome Implementation #85

Comments

ThomDietrich commented Apr 25, 2018

ThomDietrich commented Apr 25, 2018 • edited Loading

davidgraeff commented Apr 26, 2018

davidgraeff commented Apr 30, 2018

davidgraeff commented May 2, 2018

ThomDietrich commented May 2, 2018

davidgraeff commented May 2, 2018 • edited Loading

bodiroga commented May 3, 2018

davidgraeff commented May 3, 2018

bodiroga commented May 3, 2018 • edited Loading

euphi commented May 3, 2018

bodiroga commented May 3, 2018

davidgraeff commented May 3, 2018 • edited Loading

bodiroga commented May 3, 2018

davidgraeff commented May 3, 2018 • edited Loading

bodiroga commented May 3, 2018

davidgraeff commented May 3, 2018

bodiroga commented May 4, 2018

davidgraeff commented May 4, 2018 • edited Loading

bodiroga commented May 4, 2018

davidgraeff commented May 4, 2018

davidgraeff commented May 5, 2018

davidgraeff commented Oct 24, 2018

marvinroger commented Oct 24, 2018

ThomDietrich commented Apr 25, 2018 •

edited

Loading

davidgraeff commented May 2, 2018 •

edited

Loading

bodiroga commented May 3, 2018 •

edited

Loading

davidgraeff commented May 3, 2018 •

edited

Loading

davidgraeff commented May 3, 2018 •

edited

Loading

davidgraeff commented May 4, 2018 •

edited

Loading