[Announce] node-red-contrib-homie-convention

I trying to integrate the Homey to the Node-RED using the MQTT Hub app that sends Homie Convention (v3.0.1).

The problem I encounter is the Node and Property in the Node-Red Homie Node is not retained as selectable items in the drop down list. To find a new device I need to broadcast all devices again from the MQTT Hub application in Homey, then they show up, for a short while!

Additional I have another problem with the Input to the Node-Red Homie Node, the value is not updated (have checked by MQTT Explorer), is this related to above if there is no defined Node definition?

Capture

node

Hello,
thank you for using my node and sorry for the inconvenience that it is still beta.

From what you describe it looks like there is a problem with retained messages. First please take a look if MQTT Hub is sending retained messages as QoS>0 using mqtt explorer (or the tool of your choice). If not there is a solution to configure your broker of choice to handle QoS=0 messages with the retained flag set as persistent data. Take a look to this document on my github

Every restart the homie node starts with a fresh database. You should see information during the init process on the console / log. By design the Homie convention relies on the mqtt broker as a persistent database,

Hope this helps (I have to include this issue prominent in the documentation)

Chris

Thank you for guiding me...I trying to check with MQTT Explorer if the messages is retained or not, which i belive they are, or?
image

Nope you should see the orange flag on the right panel as in the screenshot
image
If you see this the QoS is not important any more. But check if the data survive a restart of your broker too.

Seems the MQTT message not is set to retained - I have open a discussion topic in the Homey forum about this. https://community.athom.com/t/mqtt-hub-gateway/6766/703?u=magnus_p

Take a look at the homie convention:

3.3 QoS and retained messages

The nature of the Homie convention makes it safe about duplicate messages, so the recommended QoS for reliability is QoS 1 . All messages MUST be sent as retained , UNLESS stated otherwise.

So if your device can’t do it you have to configure your broker to do it. Storing and saving QoS=0 Messages with retained flag is within the MQTT spec 3.3(?). The broker even „should“ keep those messages.

Now I have moved over to another MQTT Broker on my Synology NAS drive and things starting to be much better :grinning:.

I have another problem now... Everything works fine to import the data to Node-RED, the switches, buttons updates fine. But when I try to change a light status from Node-Red the correct Message is not sent. Only the set parameter is changing and anything else such the onoff property. Any Idea? I have configured switch according to below and connected this to the input

MQTT Explorer
image

Node-RED switch config
image

Hi Magnus,
It seams like the homue implentation on the Homey side is bot correct. A command is transferred (i.e. from Node-RED) via the set topic and should be executed and then acknowledged via the property topic. Either your controller is not receiving the /set command or it is not responding properly:

From the Homie convention:

Property command topic

  • homie / device ID / node ID / property ID / set : The device must subscribe to this topic if the property is settable (in case of actuators for example).

A Homie controller publishes to the set command topic with non-retained messages only.

The assigned and processed payload must be reflected by the Homie device in the property topic homie / device ID / node ID / property ID as soon as possible. This property state update not only informs other devices about the change but closes the control loop for the commanding controller, important for deterministic interaction with the client device.

To give an example: A kitchen-light device exposing the light node with a settable power property subscribes to the topic homie/kitchen-light/light/power/set for commands:

homie/kitchen-light/light/power/set ← "true"

In response the device will turn on the light and upon success update its power property state accordingly:

homie/kitchen-light/light/power → "true"

Something is a little bit strange since this configuration works fine when using ordinary MQTT in/out nodes! Now I can both receive and send the commands where the light turn on/off correctly.

I will continuing to fault trace further…

[{"id":"d6672b1f.e58f48","type":"ui_switch","z":"f6a0cce1.578a1","name":"Light Switch","label":"Light Switch","tooltip":"","group":"af4d5500.1311d8","order":9,"width":"3","height":"1","passthru":false,"decouple":"true","topic":"homie/homey-topic/lucasfonster/onoff/set","style":"","onvalue":"true","onvalueType":"bool","onicon":"","oncolor":"","offvalue":"false","offvalueType":"bool","officon":"","offcolor":"","x":610,"y":740,"wires":[["d312f4e2.7a9058"]]},{"id":"d312f4e2.7a9058","type":"mqtt out","z":"f6a0cce1.578a1","name":"MQTT Transmitter","topic":"","qos":"","retain":"","broker":"4709a5ff.a5e08c","x":830,"y":740,"wires":},{"id":"c2b2758f.cc9f78","type":"mqtt in","z":"f6a0cce1.578a1","name":"MQTT receiver light","topic":"homie/homey-topic/lucasfonster/onoff","qos":"2","datatype":"auto","broker":"4709a5ff.a5e08c","x":150,"y":740,"wires":[["83ef5d25.1bb9f"]]},{"id":"83ef5d25.1bb9f","type":"function","z":"f6a0cce1.578a1","name":"Convert String to Boolean","func":"if(msg.payload === "true"){\n msg.payload = true; \n}else{\n msg.payload = false;\n}\nreturn msg;","outputs":1,"noerr":0,"x":390,"y":740,"wires":[["d6672b1f.e58f48"]]},{"id":"af4d5500.1311d8","type":"ui_group","name":"Group 4","tab":"30048f8.763b8f","order":4,"disp":true,"width":6},{"id":"4709a5ff.a5e08c","type":"mqtt-broker","z":"","name":"magnus_p","broker":"192.168.1.189","port":"1883","clientid":"Node-Red","usetls":false,"compatmode":false,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"","willQos":"0","willPayload":""},{"id":"30048f8.763b8f","type":"ui_tab","z":"","name":"Home","icon":"dashboard","order":1,"disabled":false,"hidden":false}]

Good Morning from Berlin.

I have to dig a little bit deeper. I could not import your flow (there is something missing at the end and an error in between:

image

So perhaps you can "resend" both flows (homie version & "manual" version). You wrote that the homie/homey-topic/lucasfonster/onoff/set topic changes between true/false when you use the switch. So this should be the correct behavior in both versions.

A Boolean $datatype of the homie node should accept true/false, 0/1, "ON"/"OFF", "on"/"off", "true"/"false";

Side note: I personal use the "enum" $datatype for my DIY switches with $format "ON,OFF" because switches are normally not true or false. But this is because homie brings a high level of abstraction and do not want to be dependent on the device type. But this is more a cosmetic issue.

@Magnus_P

This is caused by not using the </> code function to insert flows. Use three back tic's (`) one a line before and after your flow and it should be able to be imported by others (see How to share code or flow json)

2 Likes

God afternoon from Gothenburg!
@Christian-Me - Finally I have now everything up and running :ok_hand: the problem was the different applications I use in the Homey and the MQTT Broker which needed to be restarted in order to be in sync.

THANK YOU - for helping me out, even if i misled you in my conclusions, there are always simpler solutions than you first expect...

By the way, any one that can share an example of how to control a dimmable light with a slider widget, Property: Dim and ON/OFF in combination

1 Like

Good afternoon from Yorkshire (UK).

Well done on this Christian, really useful to me - The password bit is working fine in latest version btw.

However I have some issues getting the dropdown list of 'devices' to be correct. I have 10 homie devices on this broker but only 6 show in the dropdown - and 3 of those are not even devices in /homie, or even homie structured - they come from higher in my topic tree. There are no homey-xxxxx devices in homie/

image

image

I have two nodes installed - the other on a different broker displays the 7 homie devices correctly. I deleted and recreated the problem node with same result.

I am getting 'device validation' errors though - I rushed a lot of the homie topics so it is almost certainly my fault. I am not sure what homie topics are used and important to you (a list would be useful) . Obviously $nodes and $properties are. Any more detailed information that these 'validation errors' could give would be most useful.

It could also be because there are 'devices' in the dropdown that are not homie devices at all of course.

Let me see what I can deduce about the validation to avoid you looking for something that isn't an issue. I do have a read access user available to my broker/homie tree if you would find that useful - it has a lot if topics (2814) within homie !

Kevin

homie is lightweight with QoS=0 and supports QoS=1 but QoS=2 is not appropriate. It's very hard work on the broker and all clients. /set messages must not be sent retained.

Hi Kevin,
Hi, fine that the password now working.
You will find a lot of information on the console to nail problems down to trace level. Just adjust your log level. The "ghost" devices are strange because the node subscribe only to homie/#. That then must be an issue in the used mqtt module (which is unlikely).

Messages about validation errors you will find when you open the device/node tree in the list. They should be marked with an question mark. Validation errors in the beginning are normal when not all data is available or not received (yet). Many problems came up when devices did not send or the broker did not handled messages as retained see link in documentation. One major problem during validation is that it is impossible to detect when all retained messages are received: MQTT determine when the last "retained" message arrived. So validation is done "on the flight". The validation is made as closely as possible to the convention (V 4.0.0). But the node should work even with some errors found. (i.e. missing $type topics).

$homie, $name, $nodes, $property, &datatype, $settable, $format (for color and enum) comes into my mind (perhaps I missed one or two)

But this could cause the "not homie" devices to show up because the node expects everything inside homie/# as a homie device.As far as I know there is no particular topic defined as an identifications, perhaps I should look for $homie. and filter everything out without that. Better is to keep your homie/# tree clean from other devices

image
image

The extensions ($fw / $stats) are not tested very deeply. So here can be a problem too.

You can find a "dump" of the homie data received by the node in global context. (if the option in extras is selected)

You can send me a dump of your broker if you like and I import them to my test broker or a link if it is available online. This helped a lot in the past. (curious to see 2814) I have only 524 :smirk:

image

QoS is not important for the node. But there is a "strange" relationship between the QoS and the retained flag in the MQTT Spec. Many "can", "should" and "as necessaries" in the MQTT spec which makes me wonder if a standard should be so vague. Worst case is that messages with QoS=0 are not treated as "retained" even if the flag is set at all.

Mosquitto by default treads these as retained as long as you don`t restart the broker or (perhaps) reach a certain threshold (100! I think). If you use mosquitto there are all necessary steps in the docs to get mosquitto behave as needed for the node..

Yes "most" messages (ALL necessary for auto-discovery) must be retained That is the persistence database of the homie convention. Be aware in mqtt explorers tree view all messages look like retained, even if they are not. Look out for the orange label (see docs too). But I think it will stay there even if the broker decides to discards QoS=0 messages "at any time" (see MQTT 3.1 / 3.1.1 spec)

As more I work with the homie convention as more I know that the people done that are quite smart and the convention is well thought out to be flexible and versatile.

I've been journeying with homie too (v3) and a lot of my issues are vagueness in the spec and of course the lack of a higher level model defining $types and how to model complete devices. To do auto recognition of a device type or capability rather than just the endpoint values requires a lot more work here.

I am currently authoring and about to release the MQTT plugin for Hubitat (a similar product to SmartThings) which both publishes devices to and imports / discovers devices using homie and the Home Assistant Discovery spec. My difficulty is trying to find homie implementations to test against. There is of course openHAB too for import into. I chatted with Harrie a lot during his Homey MQTT implementation so use that too.

Can you PM me ? I can't use that feature yet as I just registered on here. I will send you my broker login details then.

cough .. MQTT needs it though

A bit of work for me to do here... :woozy_face:
image

I've found it reliable. I think it displays this label when upon initial connection it immediately gets a payload value (so it must be retained) and then when a subsequent message comes in it removes the label - although typically the topic is still retained. If you log off and on again the retained label will return. Of course if you later publish the payload not retained it would be lost.

I don`t know what you mean with high level but for me the convention covers everything which can be read or set by and computer without defining devices in detail by the convention. But we are talking about homie and not others which might have there own pros and cons.

I was working with openhab form many years but moved to node-red because I found it easier to use a system where I know at least a little bit of the language than openHAB based on than java where I have no knowledge at all and not willing to learn in the future. And then I am totally lost and have to rely on others if something goes "not as expected" this was a long time ago :wink: https://github.com/letscontrolit/ESPEasy/issues/1314 but I learned a lot (in this thread are some interesting insides to homie and mqtt covering retained msgs and tons of topics too)
I don't know if homie is still the protocol of choice for openHAB.

mqtt explorer gives some "young players" the impression that what they see is available on the broker even it is send with retained=false and will be sent to the node when subscribing. Was just trying to narrow the possible issues. In my case the retained flag stays as long the new msg carries this flag (and with the mosquitto configured as described even when sent with QoS=0) see screenshot. If not the msg might not be retained by the broker?

Yes that is very true... they look upon it as a database.

This is what I mean, really to achieve some plug and play usage for MQTT device discovery between systems, which will be essential for widespread adoption. This may not be the roll of homie as a lower layer definition but already things like $type are used in there and not well defined.

We will need some ‘capabilities’ of devices defined as types that have mandatory sub nodes and optional ones too, so you could expose something more complex like a thermostat, AV amplifier, keypad or lock device. The presentation of enumerated values needs some standards recommended too. Maybe another higher level spec sat over homie 4 which is I agree already thorough at the individual node payload value definition.

All fun for the future..

So I'm not getting all my devices listed (6 of 10) so I guess that's to do with sizing restrictions so I kludged it by setting the root topic to
`homie/development'
to get one device 'development' which seems to work !

It has a validation error for $state at the node level in onoff but I don't think $state is required or expected there ?

image

image
image

All my device validation errors seem to be caused by $state being absent at the node level - it is present at the device level
image

I'm not sure about the name hierarchy displayed at the top either but that maybe because I kludged the root topic..

image

maybe it's the $type you're not liking ?