MQTT Homie Convention Node - questions about usage

General
1

Hi again!

As mentioned in the last of my topics, I need to better understand how to use and whether I can use the Homie Convention nodes.

Here is what I'm trying to do.

I have a load of old wireless switches and plugs, an external sensor and a wireless doorbell. All of these use 433MHz and date before cheap ESP devices :wink:. I use an RFXtrx433e USB device with Max's super excellent set of nodes to capture data and transmit commands.

What I want to do as I'm working on the next generation of my HA system is to convert all of the data I get via Max's input nodes to Homie format.

The input data looks like this:

{
  "output":"Off",
  "channel":"2",
  "status":{"rssi":4},
  "from":"lights",
  "updated":"2020-05-12T16:53:57.151Z",
  "updatedBy":"RFXtrx433e",
  "protocol":"HOMEEASY_EU",
  "id":"0x000081C0:2"
}

The topic is the ID.

I want to build a homie "device" output hierarchy for each device as it comes through with the device ID/Name based on the ID input field. Of course, I can do the reformatting of the data. I could also output the homie topics myself using a function node - but it would be much nicer if at least some of that could be done without me having to churn through a load of code myself.

Is this possible with the homie convention nodes?

I would also like to be able to grab each homie device topic hierarchy into a single object and cache that in Node-RED but that is perhaps a separate topic?

2

Ok let`s start this before I have to cook dinner :wink:

My 433 (no it is 868,3/869,525 MHz) homematic devices for example I expose to homie mqtt as this is (to some constantly increasing percentage) the center of my home automation system not Node-RED and I offload as much of "logic" into the devices and only have a command / report structure in Node-RED (in homie notation homie/device/node/property/set as command and homie/device/node/property as report) in between.

So I lately added the homie-node node (strange name but ...) and in this way all my homematic nodes appear as homie devices:

image
image960×399 43.4 KB

a converter subflow converts the data into homie properties and pass it to homie. One node per TYPE (not device) but this could be reduced to only one node because the node can be configured by JSON objects. Only the converters have to be done individually because the homematic data is not consistent between devices.

image

Somewhere else I get the data from homie and pass it to homekit for example. More complicate because the homkit node needs one node per device (or I do something wrong here)

image
image791×466 63.7 KB

or even pure "virtual devices" like my sprikler system flow is controlled via homie
image

and controlled by a simple flow which can sit on a different Node-RED instance (no link nodes here).

image
image762×376 32.6 KB

So you can define UIs for different users / devices ... or even small 3.5" touch screens with an ESP32 sending/reciving MQTT Messages (I have in the pipeline but no time to have any progress). My wall dimmer controls the ceiling fixture but also sends messages to WIFI lightbulbs in the floor lamp via mqtt homie directly (really low latency here). Node-RED only "listens" and updates the dashboard. Only when a Node-RED wants something else it send own /set messages.
Another example is my valve for filling my pond. It starts and stop the pump automatically without Node RED involved. The timer is included inside the ESP. This is all what Node-RED needs to do/know.

image

Another feature of homie are the retained messages. If my ESPs in some light bulbs crash they will get their values after reboot and come back in the state they where set before. Even a offline device will get its values changed in the meantime when it reconnects. Restart of Node-RED is no problem too because all retained messages set everything to the last know values. And if a different device changes the value all "subscribers" will update there UIs or can react accordingly.

Yes I think so. Perhaps there are a few bugs left so better install the node from my github instead of npm for faster updates. I develop this on my live system so it is in practical use right now.

My complete flows you can find on github too (be carefull everything in a state of unfinished)

3

Hi again Christian, thanks for responding.

This is where we differ. I want 1 node per device & that seems to be the generally accepted use-case for Homie. With all of the properties of that device suitably listed.

This gives me a set of electronic virtual devices that map to real-world devices. I will use standard naming conventions for nodes in a device:

devices                  
     deviceid1            
         switch-n         
              power       
              last_updated
              ...         
         environment      
              temperature 
              humidity    
              pressure    
              light       
              sound       
              ...         
         info             
              location    
              altitude    
              geoloc      
              description 
              ...         
     deviceid2            
     ...

So how do I configure one of your nodes to be able to write to a device - devices/mynewdevice/... without having to have defined the device first since - in your published release anyway - the only node that can write to homie topics is via the homie-node node and that uses the "broker" configuration which is where the device is pre-configured.

What's more, that "device" as you say is one device for the whole of Node-RED which can't work for me.

It also makes a few odd assumptions about the attributes such as the $mac address which pics up an IP address and the $localip which strangely picks up the address of docker and not my actual LAN address. $implementation should, I think be the controller implementation rather than just "x64"?

I get this, it isn't an issue, I've done this with both homie and other forms of hierarchy.

I'll try and take a look at those.

4

@TotallyInformation, I converted all my home stuff to ‘homie-aligned’ 16mnths ago, still building. I am busy documenting it on https://github.com/IoTPlay/homedig2 . I have not built Node-RED node/s yet, all in flows & funtions. I now are able to represent all my IoT stuff in global context in homie standard, but have added to the definition. (Homie allows this, if done right - I have not yet published back to them). Homie std sorted out lots of gremlins to get things to standard.

I register all addt info (some devices too old to convert to support all of homie std, including retained) in a yaml config file, which NR reads at startup, and further adds to the homie storage on global context. I have also last 3 weeks added the ability to read CoAP devices periodically, then feed the received http json back into the homie structure. (Shellies has a gr8 CoAP api).

Homie std relieved my architecture from lots of issues, and drove standardisation. Will also be blogging more about it on https://iotplay.org

1 Like
5

First - should be possible with some modifications of the homie-node.

Second - lets define device, node and property. A device is a physical device represented by the mqtt connection. The device should subscribe to <baseTopic>/<deviceName>/+/+/set to receive only commands. The node is a functional unit like a sensor let’s say a BME280 or a thermostat (see first page of the convention). And a node has one or many properties like temperature/humidity/pressure or setTemperature/currentTemperature. So the overall supercar device example has many nodes some are unique others are used many times like the thermostat for the driver and passenger side. Or an ESP8266 (in my Greenhouse) with an BME260, a lux-sensor, a relay for the sprinkler system, 4 relays for garden illumination. They all could be connected to separate ESPs but this makes no sense as they are all located at the same place / case / power supply.

But I think the homie convention is in this case not strict - let’s say flexible. So I had the definition of a physical device or controller like Node-RED taken as one device in mind.

I think we could agree to a device have a network connection, IP, cpu, power supply, battery in common.But then you will end with more than 1000 topics as every device have many "overhead" topics, especially then you like to use all 3.0.1 or 4.0.0 with extension ($fw / $stats):

image
image732×304 10.1 KB

But certainly currently for example the battery state of a thermostat is not included unless you "translate" it into a property.
So I agree that my "translated" devices like a thermostat is not implemented as 100% homie device. More like a device wired to Node-RED.

The configuration JSON currently looks like:

{
    "homie": {
        "$properties": {
            "state":{
                "$name": "program state",
                "$datatype": "enum",
                "$settable": false,
                "$retained": true,
                "$format": "stop,prepareing,runing,error"
            }
        }
    }
}

but in your (more complete) approach it will look like this (some $ properties are missing here)

{
    "baseTopic": {
        "deviceName": {
            "$name": "Demo Device",
            "$nodes": {
                "firstNode": {
                    "$name": "Demo Node",
                    "$properties": {
                        "state": {
                            "$name": "program state",
                            "$datatype": "enum",
                            "$settable": false,
                            "$retained": true,
                            "$format": "stop,prepareing,runing,error"
                        }
                    }
                }
            }
        }
    }
}

I think I was not clear. I only mentioned the homekit nodes where one Flow-Node is needed per homeKit node. This is not flexible and not nessesary for homie-device nodes.

Just to be clear, for me is a:

  • a translated device a physical device (own CPU) translated through Node-RED (like a homematic thermostat or the devices you want to transform to homie)
  • a virtual device is a device which is actually some kind of logic in node red exposed to homie (like my sprinkler system. No own CPU

image
image760×344 56.6 KB

In the end the broker node has to subscribe to +/+/$homie and individually to many detected base topics.
All possible but it will take some afford = time. Certainly willing to do this. Will make the remote device table for my own system simpler again.

I will have a lot of questions, for example:

  • who is generating the $ready updates in behalf of the translated device?
  • the LWT mechanism has to be "simulated"

What do you think?

6

As for now I didn`t felt the need to add anything to my nodes as all fit's perfectly.

My main issue working with the homie convention is more mqtt related: timing. As you can't predict the order messages arrive and what makes things more difficult to determine when you have received the last = all retained messages.

But over all I`m very happy I made the move to standardize my communication. I'm not too strict what is allowed or not especially after a difficult conversation how QoS=0 messages are traded by OpemHAB build in broker rendering most of the homebrew devices unable to send QoS>0 messages "undetected".

a few days ago we finaly decided for the new Banner:

Works with@2x

2 Likes
7

I also have several cases where I could not find a place in Homie for an attribute - but tried ('roughly' ?) to stick to the guidelines provided in the Homie Extentions - and maybe onde day I will publish them back to Homie - and influence the next releases. For instance- I am storing the javascript timestamp of info I receive - so I can do time calcs with them, not required if you have Retained mqtt enabled for all, but as I said I also store CoAP received periodic info into the same structure. So I need it - and the standard does not have it - and many other things. @TotallyInformation - if their is interest in this - plaus how I am storing info from Homie structure -and have a common model to retrieve the infor for BI purposes (still have some bugs) - I will publish more -and code.

8

That is already implemented (see screenshot in the other thread). And basic homie validation - still not perfect but some good hints.

I’m happy that the homie ball starts rolling and happy for all input.

9

Not too different for me:

  • ESP devices that can talk homie directly
  • ESP devices that are using firmware that doesn't directly support homie - translated via NR
  • Non-ESP devices (mainly connected via serial) - translated via NR

This is indeed an issue. I think that ideally, a home device node would create a new broker connection for every new device. But of course, that means that the system needs to know about existing devices.

Possibly a Node-RED "device" should be at the root as now and that should be used to discover all of the existing homie devices on startup. That would ensure an up-to-date list.

This does, however, assume that it is possible to create a new broker connection on the fly which none of the current MQTT nodes can do.

A fudge workaround that would be fairly easy to do would be to put the logic into Node-RED. The node.on('close', cb) can be used to deal with Node-RED closing down, nodes being removed and so on. It wouldn't be perfect since there would still be occasions when a crash of NR would leave the wrong status but it would cover most things. Might be good to implement in the short term while trying to work out a better long-term solution?

10

Actually, this is a weakness of the Homie convention. The lack of last updated timestamps as a standard attribute/property. Node types such as switches need to track their last timestamp very often. With no $updated attribute or similar, you can only really define a switch as a node and not as a property as might be desired in some cases. At least without defining a custom attribute extension.

Also the assumption that homie "devices" will always be sending a keepalive heartbeat of some kind which is not desired for low-power use nor for intermittent devices such as switches. This would be better served by having a controller "device" as standard where the controller is sending a heartbeat and allowing the other devices to have optional rather than required dynamic $state.

I think the other potential weakness is not having some better guidance for common nodes/properties. For example, I think that an info node should be common where you can define additional metadata about the device such as its location and a description. It would also be really good if people knew their was a common standard for sensors - I'm using a node called environment for example that contains all of the environmental sensor properties from a device (temperature, humidity, atmos pressure, lightm sound, etc). Having a standard for switches would also be really useful since this will be a very common use-case.

Standards like these would be of massive benefit to all and would undoubtedly help with adoption of the Homie convention over time.

11

Agree. I will respond to further to your post above. But on related topic - as you know from other posts under Proejcts I made probably a year +, this project I have - on Github (not the cod yet), has had a previous incarnation - I build my own standard - before I kno abotu Homie - a Yaml file that also configures all. For a year - as I added devices (including RF) - the wheels started coming off the standard I thought would work. I then even included some concepts of Apple's approach (although the data model is far too complex for one person to replicate) Then I learnt about Homie - and its pretty dam close to what I need - specially with the way I have extended it now.

The thing that took me longest - was to translate the interactions - into storage into global context. I unfortunately completed it before @Christian-Me published his nodes - maybe one day I stil will look if I can replace mine with his.

I must still do a write-up on the data model - but here is a quick view @ it - and the best way I could do internal storage with a big json mapping of all of it + my extentions - so that I can interact with it from different inputs - both mqtt and CoAP and others, and Report from it - and maybe it was my limited knowledge of how to interact with a json mixed arrays and objects - but so be it - it works.

Above is what I did - and what my internal storage looks like - could also publish a sample json. I now have objects which creates from yaml files, or from homie mqtt, updates via mqtt, or CoAP, and push out BI to an http RESTful endpoint - I am starting to use these for other purposes. Nicely standadised - easy and fast to implement new funcitonality.

12

Thanks for responding. I agree that Homie is close and I'd really like to use it and not abuse it too much which is why these discussions are important.

As an (incomplete) example of what I'm trying to get to, here is the output from my next-gen sensor platform using esp homie:

image
image629×1005 22 KB
image
image635×731 16.2 KB

And here is a quick mock-up I've done manually of a conversion from my RFXtrx433e switch inputs:

image
image422×698 14.3 KB

Obviously not everything included in that as yet as I quickly mocked it up using a function node. At the moment, it sends the whole device hierarchy out every time it gets an input msg. However, that is just detail - retaining existing device ID's and only sending some things on the first output or on change.

You can see that not everything is identical. For example there are some extras under $implementation that capture where the data came from and what type of source it was.

But you can also see how I'm trying to standardise the nodes in use.

I am going to stop using ESPeasy and other frameworks if I can get all of this working because they never quite do what I want, using my own bespoke code and trying to standardise the data structures gives me a much better and more flexible platform and lets me understand more when things go wrong. That isn't to say that I want to do everything from scratch though since I simply don't have anywhere near enough time. I will likely continue to use Tasmota for SONOFF and other commercial devices but that is about it.

13

Hi,
I know there might be some "things missing" in the homie convention but i followed the development for more than a year now. First I was arguing too but in the end all works much better than I expected in the beginning. In my opinion the convention is open enough to do whatever you like (or in the end either participate or do your own extension).

So I took your suggestions and made a first test.
First I checked how to auto discover the Base Topic (and got the naming right).

image
image454×537 14.8 KB

in auto discover mode the broker node subscribes to +/+/$homie and if it receives a message it will then subscribe to this Base Topic.

13 May 22:20:26 - [info] [homie.device.config:mosquitto@home] New Homie base topic detected: 'homie' $homie=4.0.0 subscribed to homie/#
13 May 22:20:26 - [info] [homie.device.config:mosquitto@home] New Homie base topic detected: 'devices' $homie=4.0.0 subscribed to devices/#

So this was a easy one. Now I have to update all my code to reflect the extra level in the data structure. I thought about all this for a while and decided to rewrite the broker node (and the homie-device, homie state and homie-node) instead of patching code which is already full of patches. This time based on definition objects instead of switch/case and if/then structures. That would make updates easier later. Here is the homie convention as I read it in a nutshell

    this.homieConvention = {
      "deviceId": {"type":"string", "required":true},
      "$homie": {"type":"string", "required":true},
      "$name": {"type":"string", "required":true},
      "$state": {"type":"enum", "required":true, "default":"lost", "format":"init,ready,disconnected,sleeping,lost,alert"},
      "$nodes": {"type":"array", "required":true, "default":"string"},
      "$extensions": {"type":"string", "required":true, "default":""},
      "$implementation": {"type":"string", "required":false},
      "_nodes":{
        "nodeId": {"type":"string", "required":true},
        "$name": {"type":"string", "required":true, "default":""},
        "$type": {"type":"string", "required":true, "default":""},
        "$properties": {"type":"array", "required":true, "default":""},
        "_properties": {
          "propertyId":{"type":"string", "required":true},
          "$name":{"type":"string", "required":true},
          "$datatype":{"type":"enum", "required":true, "default":"string", "format":"string,integer,float,boolean,enum,color,DateTime,Duration"},
          "$format":{"type":"string", "required":false},
          "$settable":{"type":"boolean", "required":false, "default":false},
          "$retained":{"type":"boolean", "required":false, "default":true},
          "$unit":{"type":"string", "required":false, "default":""}
        }        
      }
    }

Perhaps you see something I missed or got wrong

Same for extensions (which I noted could be addable in the future). I added some extra properties like suitable UI-Icons to have the opportunity to be flexible with future extensions perhaps even user defined ones.

    this.homieExDef = {
      "$state":{"icon":"fa fa-bolt","type":"enum","unit":"","required":true,"default":"lost","format":
        {
          "init":"fa fa-cog fa-spin",
          "ready":"fa fa-spinner fa-spin",
          "disconnected":"fa fa-times",
          "sleeping":"fa fa-moon-o",
          "lost":"fa fa-question-circle",
          "alert":"fa fa-exclamation-triangle"
        }
      },
      "$homie":{"icon":"fa fa-label","type":"string","unit":"","required":true,"default":"n/a","format":""},
      "$nodes":{"icon":"fa fa-labels","type":"array","unit":"","required":true,"default":"n/a","format":""},
      "$extensions":{"icon":"fa fa-labels","type":"string","unit":"","required":true,"default":"n/a","format":""},
      "$type":{"icon":"fa fa-cogs","type":"string","unit":"","required":true,"default":"n/a","format":""},
      "$implementation":{"icon":"fa fa-code","type":"string","unit":"","required":false,"default":"n/a","format":""},
      "$localip":{"icon":"fa fa-address-card-o","type":"string","unit":"","required":true,"default":"n/a","format":""},
      "$mac":{"icon":"fa fa-barcode","type":"string","unit":"","required":true,"default":"n/a","format":""},
      "$fw":{
        "name":{"icon":"fa fa-file-code-o","type":"string","unit":"","required":true,"default":"n/a","format":""},
        "version":{"icon":"fa fa-code-fork","type":"string","unit":"","required":true,"default":"n/a","format":""},
        "*":{"icon":"fa fa-label","type":"string","unit":"","required":false,"default":"","format":""} 
      },
      "$stats": {
        "interval":{"icon":"fa fa-undo","type":"integer","unit":"sec","required":true,"default":0,"format":""},
        "uptime":{"icon":"fa fa-clock-o","type":"integer","unit":"sec","required":true,"default":0,"format":""},
        "signal":{"icon":"fa fa-wifi","type":"integer","unit":"%","required":false,"default":0,"format":"0:100"},
        "cputemp":{"icon":"fa fa-thermometer-half","type":"float","unit":"°C","required":false,"default":0,"format":""},
        "cpuload":{"icon":"fa fa-microchip","type":"integer","unit":"%","required":false,"default":0,"format":"0:100"},
        "battery":{"icon":"fa fa-battery-half","type":"integer","unit":"%","required":false,"default":0,"format":"0:100"},
        "freeheap":{"icon":"fa fa-braille","type":"integer","unit":"bytes","required":false,"default":0,"format":""},
        "supply":{"icon":"fa fa-plug","type":"float","unit":"V","required":false,"default":0,"format":""},
      }
    };

This is already in use in the home.state node

image
image698×853 27 KB

Last but not least ... the Base Topic is not part of the homie Topology So fixing it to a single user defined one I thought was not wrong in the beginning. But I got the argument to have a extra level of organisation could be convenient. What I got out of many conversations on github is that many "feature requests" you mentioned where already discussed. It is a good lecture for understanding the convention and it's goals better. The only thing I really miss in many conversations about the homie convention is a "theory of operation" document to explain the "why" better. Why messages have to be retained or the logic behind the /set topic. With this my ESPs (running ESPEasy with the Homie plugins - far more developed than the currently distributed - when I will find time to rebase my repro and do a PR?). I love how easy it is to "survive" a wd-reset without RTC-Memory or wearing out the flash memory when using the homie convention.
Additionally I love that the convention do not have to many "should", "may" or "cans" - I was really shocked about all the subjunctive forms in the MQTT standard (especially around retained messages). As an architect I could say that many buildings will collapse if our standards would be so vague :slight_smile:

That's enough for today.

14

You could you send me the flow/node for further tests during developement. I did not read everything in detail but perhaps take a look at:

  • $datatype = date is not correct. there is a DateTime payload already defined - and $format don't know "ISO"
  • For location there is or was a issue/discussion on github. Don`t know if this is closed.
  • $implementation do not have "sub topics" - something for an extension?
  • $name should be a human readable name

I read this :slight_smile:

15

MQTT works with only a single connection. The subscribed topics can be changed "on the fly"

Ok a last one to clear the dockets. During development (even on my live system) I CTRL-C node read many times. No problem. The retained messages magically set the state of everything (I put on the broker) to the last known state. Even if the state changed during downtime. So every parameter I use when is related to homie properties is stored (if necessary) "memory only". This is a very convenient "replay" mechanism. No need to worry - not on the controller side and not on the device side too.

16

Wow, great work Christian.

Can't see anything that you haven't covered in the extensions.

But unfortunately, unlikely to be accessible to newcomers. Really, any of that discussion should be summarised into FAQ's so that it is discoverable and accessible.

The problem is that the convention is complex enough to present a considerable learning curve to newcomers. That friction restricts adoption.

Actually I have no problem with those though the /set part wasn't particularly obvious in the docs I didn't feel and I've only recently discovered it. Probably not helped by the fact that the convention has moved on a long way and I'd not looked at it for a long time so many assumptions I was making were outdated.

Also, many of the tools are not being updated to match the convention. Like the esp homie library which is still on v3.

I won't use plugins with something like ESPEasy unless someone is regularly rebuilding the whole thing, keeping it in sync with the many core updates. I'm not a strong C/C++ programmer and I only use the Arduino IDE. I also don't have the time.

That's why I so often end up back with my own code. Though it takes me longer to get started, it means I understand what is going on and I get exactly what I want.

If including Homie in ESPEasy was simpler, I would certainly have used it. But even then, it probably wouldn't let me tweak it? Does it include update timestamps for switch events for example?

I will.

Ah well that was just something that I did quickly so I didn't pay much attention to it. I only stuck the format in to remind me to be more specific should I take the flow forwards.

It should have said "ISO8601" really which is the date/time common ISO format: "2020-05-14T13:20:00.000Z" as used by the JavaScript toISOstring() function and commonly used in IT.

I will try to have a look. There are 2 types of location really. One is descriptive and the other is geographic and I will want to have both. Though descriptive (e.g. "Office", "Bathroom") is the most useful for most of what I'm doing. Altitude is also needed in order to make sense of atmospheric pressure sensors since you really want to be publishing sea-level equivalent pressure since this is what is used in weather forecast.

Well it does in the esp homie library which is what created that. So I mirrored that in my RFX->Homie flow. It could be moved to the info node.

What's wrong with "0x000081C0:2"? :grin:
Actually, some devices might need human readable names but others don't. In fact, I could rename them to their location (with a number to avoid clashes as some rooms will have >1 device). It is just that the name would then need to be settable for when the device gets moved which they do sometimes.

Not quite. Since Homie convention wants each device to have the $status dynamically update, you should really use a different connection for each so that the LWT kicks in where needed to reliably take the device status offline. This is exactly what happens when the MQTT topics for a device are being pushed from the device. It is only where you are using a different service to output multiple devices that this gets confusing - as with Node-RED. Ideally, your homie output node would allow the dynamic creation of a new broker connection for each new device so that the device list can be data driven rather than fixed. The use-case here being me adding new 433MHz devices which the RFXtrx433E will automatically discover and output data for and the homie node would then create the new homie device.

That leaves aside some questions about device metadata but I already have a previous solution for that where I store that data in a retained NR global variable. I have a uibuilder page that lets me add any required extra data such as the descriptive location. That can then be merged with the RFX data.

It is, but it doesn't work where Node-RED is the output for the homie part but not the input as is the case with my RFX inputs. The status changes independently of Node-RED but will only be reflected in homie if Node-RED (and the RFX) is running correctly. Once Node-RED is being used, it needs to also retain knowledge of all of the existing nodes if you want (as I do) to reduce the traffic and only output the more "fixed" parts of the homie convention once.

Since homie makes all topics retained anyway, this isn't a big issue (there is a potential small timing issue when Node-RED starts up). Node-RED would receive all of the homie data on startup as long as it was wild-card subscribed to the root topic.

We have to remember that we have two completely different homie workflows here.

  1. Direct device outputs - not a problem, work as expected
  2. Indirect device outputs via Node-RED - an unknown number of devices will flow through Node-RED with it translating from input data to homie data.
17

Here is the example flow:

[{"id":"eaaf8a1c.a6cc58","type":"mqtt in","z":"956152e4.d7617","name":"RFX/# from local","topic":"RFX/#","qos":"2","datatype":"json","broker":"3784c9f0.57bab6","x":150,"y":320,"wires":[["b0f0222c.5d28a","336cefc2.adb9a"]]},{"id":"917c97e7.dd7dc8","type":"debug","z":"956152e4.d7617","name":"RFX->Homie","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":630,"y":320,"wires":[]},{"id":"b0f0222c.5d28a","type":"function","z":"956152e4.d7617","name":"Convert to Homie","func":"// {\"output\":\"Off\",\"channel\":\"2\",\"status\":{\"rssi\":5},\"from\":\"lights\",\"updated\":\"2020-05-11T18:59:41.368Z\",\"updatedBy\":\"RFXtrx433e\",\"protocol\":\"AC\",\"id\":\"0x0000A9F8:2\"}\n\nconst rfxin = msg.payload\nconst rootTopic = 'homie'\n\nlet topic = ''; let value = ''\n\n// Get the cached RFX data\nconst rfx = global.get('rfx', 'file') || {}\n\n// If current ID not in RFX, then treat as new and send ALL data - note that we should enhance this to only send differences\nif ( ! rfx[ rfxin ] ) {\n    \n}\n\ntopic = `${rootTopic}/${rfxin.id}`\n\n// Some of these SHOULD only be sent once - needs a table of existing nodes\nnode.send({'topic':`${topic}/$state`, 'payload': `ready`})\nnode.send({'topic':`${topic}/$homie`, 'payload': `4.0.0`})\nnode.send({'topic':`${topic}/$name`, 'payload': rfxin.id})\n//node.send({'topic':`${topic}/$mac`, 'payload': ``})\n//node.send({'topic':`${topic}/$localip`, 'payload': ``})\nnode.send({'topic':`${topic}/$stats/signal`, 'payload': rfxin.status.rssi})\n//node.send({'topic':`${topic}/$fw/name`, 'payload': ``})\n//node.send({'topic':`${topic}/$fw/version`, 'payload': ``})\nnode.send({'topic':`${topic}/$implementation`, 'payload': `node-red-rfx`})\nnode.send({'topic':`${topic}/$implementation/version`, 'payload': ``})\nnode.send({'topic':`${topic}/$implementation/source`, 'payload': rfxin.updatedBy})\nnode.send({'topic':`${topic}/$implementation/protocol`, 'payload': rfxin.protocol})\nnode.send({'topic':`${topic}/$implementation/type`, 'payload': rfxin.from})\n\nlet hNodes = []\n\n{   // Info node\n    let hNode = 'info'; hNodes.push(hNode)\n    let props = []; let prop = ''\n    node.send({'topic':`${topic}/${hNode}/$name`, 'payload': `Information`})\n    node.send({'topic':`${topic}/${hNode}/$type`, 'payload': `Info`})\n    \n    prop = 'location'; props.push(prop)\n    node.send({'topic':`${topic}/${hNode}/${prop}`, 'payload': ``}) // Needs lookup info\n    node.send({'topic':`${topic}/${hNode}/${prop}/$name`, 'payload': `Location`})\n    node.send({'topic':`${topic}/${hNode}/${prop}/$datatype`, 'payload': `string`})\n    node.send({'topic':`${topic}/${hNode}/${prop}/unit`, 'payload': `descriptive`})\n    \n    node.send({'topic':`${topic}/${hNode}/$properties`, 'payload': props.join(',')}) // description,altitude,geoloc, switch_n\n}\n\n{   // Switch node\n    let hNode = 'switch_1'; hNodes.push(hNode)\n    let props = []; let prop = ''\n\n    node.send({'topic':`${topic}/${hNode}/$name`, 'payload': `Output`})\n    node.send({'topic':`${topic}/${hNode}/$type`, 'payload': `switch`})\n    \n    prop = 'power'; props.push(prop)\n    node.send({'topic':`${topic}/${hNode}/${prop}`, 'payload': rfxin.output})\n    node.send({'topic':`${topic}/${hNode}/${prop}/$name`, 'payload': `Power`})\n    node.send({'topic':`${topic}/${hNode}/${prop}/$datatype`, 'payload': `enum`})\n    node.send({'topic':`${topic}/${hNode}/${prop}/$format`, 'payload': `On,Off`})\n\n    prop = 'updated'; props.push(prop)\n    node.send({'topic':`${topic}/${hNode}/${prop}`, 'payload': rfxin.updated})\n    node.send({'topic':`${topic}/${hNode}/${prop}/$name`, 'payload': `Last Update`})\n    node.send({'topic':`${topic}/${hNode}/${prop}/$datatype`, 'payload': `date`})\n    node.send({'topic':`${topic}/${hNode}/${prop}/$format`, 'payload': `ISO`})\n\n    node.send({'topic':`${topic}/${hNode}/$properties`, 'payload': props.join(',')})\n}\n\nnode.send({'topic':`${topic}/$nodes`, 'payload': hNodes.join(',')})\n","outputs":1,"noerr":0,"x":390,"y":320,"wires":[["917c97e7.dd7dc8","49ba0661.2d40f8"]]},{"id":"336cefc2.adb9a","type":"debug","z":"956152e4.d7617","name":"Raw RFX/#","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":370,"y":280,"wires":[]},{"id":"49ba0661.2d40f8","type":"mqtt out","z":"956152e4.d7617","name":"","topic":"","qos":"","retain":"","broker":"3784c9f0.57bab6","x":610,"y":260,"wires":[]},{"id":"3784c9f0.57bab6","type":"mqtt-broker","z":"","name":"nrmain-local","broker":"localhost","port":"1883","clientid":"nrmain-local","usetls":false,"compatmode":false,"keepalive":"60","cleansession":true,"birthTopic":"services/nrmain","birthQos":"0","birthRetain":"true","birthPayload":"Online","closeTopic":"services/nrmain","closeQos":"0","closeRetain":"true","closePayload":"Offline","willTopic":"services/nrmain","willQos":"0","willRetain":"true","willPayload":"Offline"}]

An example of the input data payload is in the comments at the start of the function node.

For clarity and ease of reading, here is the function:

// {"output":"Off","channel":"2","status":{"rssi":5},"from":"lights","updated":"2020-05-11T18:59:41.368Z","updatedBy":"RFXtrx433e","protocol":"AC","id":"0x0000A9F8:2"}

const rfxin = msg.payload
const rootTopic = 'homie'

let topic = ''; let value = ''

// Get the cached RFX data
const rfx = global.get('rfx', 'file') || {}

// If current ID not in RFX, then treat as new and send ALL data - note that we should enhance this to only send differences
if ( ! rfx[ rfxin ] ) {
    
}

topic = `${rootTopic}/${rfxin.id}`

// Some of these SHOULD only be sent once - needs a table of existing nodes
node.send({'topic':`${topic}/$state`, 'payload': `ready`})
node.send({'topic':`${topic}/$homie`, 'payload': `4.0.0`})
node.send({'topic':`${topic}/$name`, 'payload': rfxin.id})
//node.send({'topic':`${topic}/$mac`, 'payload': ``})
//node.send({'topic':`${topic}/$localip`, 'payload': ``})
node.send({'topic':`${topic}/$stats/signal`, 'payload': rfxin.status.rssi})
//node.send({'topic':`${topic}/$fw/name`, 'payload': ``})
//node.send({'topic':`${topic}/$fw/version`, 'payload': ``})
node.send({'topic':`${topic}/$implementation`, 'payload': `node-red-rfx`})
node.send({'topic':`${topic}/$implementation/version`, 'payload': ``})
node.send({'topic':`${topic}/$implementation/source`, 'payload': rfxin.updatedBy})
node.send({'topic':`${topic}/$implementation/protocol`, 'payload': rfxin.protocol})
node.send({'topic':`${topic}/$implementation/type`, 'payload': rfxin.from})

let hNodes = []

{   // Info node
    let hNode = 'info'; hNodes.push(hNode)
    let props = []; let prop = ''
    node.send({'topic':`${topic}/${hNode}/$name`, 'payload': `Information`})
    node.send({'topic':`${topic}/${hNode}/$type`, 'payload': `Info`})
    
    prop = 'location'; props.push(prop)
    node.send({'topic':`${topic}/${hNode}/${prop}`, 'payload': ``}) // Needs lookup info
    node.send({'topic':`${topic}/${hNode}/${prop}/$name`, 'payload': `Location`})
    node.send({'topic':`${topic}/${hNode}/${prop}/$datatype`, 'payload': `string`})
    node.send({'topic':`${topic}/${hNode}/${prop}/unit`, 'payload': `descriptive`})
    
    node.send({'topic':`${topic}/${hNode}/$properties`, 'payload': props.join(',')}) // description,altitude,geoloc, switch_n
}

{   // Switch node
    let hNode = 'switch_1'; hNodes.push(hNode)
    let props = []; let prop = ''

    node.send({'topic':`${topic}/${hNode}/$name`, 'payload': `Output`})
    node.send({'topic':`${topic}/${hNode}/$type`, 'payload': `switch`})
    
    prop = 'power'; props.push(prop)
    node.send({'topic':`${topic}/${hNode}/${prop}`, 'payload': rfxin.output})
    node.send({'topic':`${topic}/${hNode}/${prop}/$name`, 'payload': `Power`})
    node.send({'topic':`${topic}/${hNode}/${prop}/$datatype`, 'payload': `enum`})
    node.send({'topic':`${topic}/${hNode}/${prop}/$format`, 'payload': `On,Off`})

    prop = 'updated'; props.push(prop)
    node.send({'topic':`${topic}/${hNode}/${prop}`, 'payload': rfxin.updated})
    node.send({'topic':`${topic}/${hNode}/${prop}/$name`, 'payload': `Last Update`})
    node.send({'topic':`${topic}/${hNode}/${prop}/$datatype`, 'payload': `date`})
    node.send({'topic':`${topic}/${hNode}/${prop}/$format`, 'payload': `ISO`})

    node.send({'topic':`${topic}/${hNode}/$properties`, 'payload': props.join(',')})
}

node.send({'topic':`${topic}/$nodes`, 'payload': hNodes.join(',')})

As mentioned, this is just a proof of concept and far from complete.

18

Right even if the documentation is quite precise some Background docs could be helpfull. As usual PRs appreciated.

Don`t like that, I know that because I used the library for my christmas tree project too. But never mind, a good example to test if a) the homie node handles these cases and b) be expandable.

ESPEasy is based on plugins isn't it. The homie plugins are part of the development builds. I didn't got too much feedback (neither positive or issues) so I did further improvements without a new PR - witch is sitting here for ages - only have to pull/rebase my fork ... not a quick task if you fall so far behind. But I agree again homebrew firmware is much more versatile - but don`t help newommers.

Hmmm I understand ... Don`t think that is a good approach. For virtual or translated devices the homie node could (and should) take care of setting the $state to lost on behalf of the remote device. OK you loose the capability to set all $states to lost if Node-RED crashes but I think it is a good compromise instead of opening many http connections and instances of the library and message handlers. But I take a look, perhaps I can do this as an option.

Have to think about it.
I don't get the controller / device role at all. For me there is or should not be a difference. Some Examples.
A) Sensor device (Envioment) - conroller read/display/process the data OK, clear.
B) Actor (Say light bulb with $setable parameters) + plus a movement sensor. Who is the controller or device?. The moment sensor should (or could) send a /set command directly to the lightbulb. The actor confirms by setting the property. If the actor don't want to have this state "persistent" (like if a timer runs out) it clears the /set topic and sets the property to the correct state after timeout. Who is the controller or device? Or better why make a difference at all? In this case Node-RED sits beside and listens (perhaps updates the UI) and only if it wants to take control it emits a /set command again same as the movement sensor. But does this makes it to a controller? Is the movement detector a controller & a device? But this are only words.
C) OK a controller "should" do auto-discovery and all that but that`s not a must.

That is what the homie-node is designed for. Play the role of a homie device/node ... on behalf of n-CPUs doing something.

Enough theory. Next step is to rewrite the homieTree and the UI to have this extra level in the hierarchy or better update the homie-node node to get actual live data on different base topics and "virtual" devices? And only patch the homie-device node to send to individual base topics as receiving already works.

19

Well, if I can ever learn enough to do that, I will. Unfortunately, I seem to be a long way off as yet.

Don't really think you could say it is based off them. You can use them. But then I think you have to recompile your own library each time there is an update to either ESPeasy itself or to the plugin. If I'm doing that, I might as well write my own custom code and be done with it.

That may be because of the above. I certainly looked at your plugin but dismissed it simply because it looked like a lot of trouble to go to in order to compile a custom version - not a comment on your code of course.

Perhaps I am mistaken? Is it in fact easy? Can it be done with Arduino IDE? How easy is it to update ESPeasy itself when a plugin is in use?

But it is the homie way. And it would be the most robust.

That was what I suggested as a workaround. It may well be good enough. If you do it this way, as I said, probably the best idea is to create a node-red homie "device" that has its own connection as you do at the moment, its just that you would add the ability to create devices dynamically that will use the pseudo $state processing. That is what I'd do if I were setting this up with core nodes only.

There wouldn't be a difference if you had 1 MQTT connection for each virtual device in Node-RED Homie :wink:

Better to consider 2 sets of switches rather than confusing things with a sensor and a switch.

Imagine a switch attached to an ESP. The controller is the ESP device and it connects directly to MQTT and sends data as needed. When I say a switch here, I really am thinking about the switch - e.g. a button or a touch sensor, I'm not thinking about a relay. Effectively the switch is a special form of sensor.

Another switch is attached to Node-RED (via serial lets say, or directly connected to a Pi). The controller is now Node-RED but currently you only have a single MQTT connection shared across multiple "devices". You could, as you have, say that Node-RED is the "device" and that all sensors hang off it. That does, of course work, but you are tightly coupling the devices to Node-RED and Node-RED to the Homie topics.

A more flexible and less tightly coupled architecture is to treat Node-RED as a virtual controller capable of pretending to be multiple real controllers (like a VM host). That way, it becomes a matter of configuration rather than re-engineering if you want to move a real device to a different Node-RED host or to an ESP. Your overall Homie hierarchy would stay the same and your device list would stay the same, only a couple of attributes might change (maybe the localip for example). In turn, none of your other processing or control logic changes at all and neither do any dashboards or historic data gathering via InfluxDB.

Hope that is a better explanation than I've previously given?

To summarise. What you have is logical enough with a single parent "device", it just isn't flexible enough for what I'm envisioning. It makes your homie controller a lot simpler for sure but at the cost of later flexibility and potential rework.

20

Hi Julian @TotallyInformation ,

Can you please help me with formatting my data in function node to convert it to Homie like you did here. There are 2 types of messages (split by sensors and status and unsplit with topic name sensorValues and deviceStatus) from my devices. Following is format of my MQTT messages:

Published topic 'sensorValues with payload {"location":"Bedroom1","voltage":"2.62","level":"8","presence":"0","motion":"0","custom":"71"}
Published topic 'Bedroom1/voltage with payload 2.62
Published topic 'Bedroom1/level with payload 8
Published topic 'Bedroom1/presence with payload 0
Published topic 'Bedroom1/motion with payload 0
Published topic 'Bedroom1/custom with payload 71
Published topic 'DeviceStatus with payload {"location":"Bedroom1","rssi":"-65","mode":"0","ip":"26","channel":"7","sleeptime":"1","uptime":"151"}
Published topic 'Bedroom1/rssi with payload -65
Published topic 'Bedroom1/mode with payload 0
Published topic 'Bedroom1/ip with payload 26
Published topic 'Bedroom1/channel with payload 7
Published topic 'Bedroom1/sleeptime with payload 15
Published topic 'Bedroom1/uptime with payload 151
Published topic 'sensorValues with payload {"location":"Livingroom","voltage":"2.62","temperature":"75","humidity":"48","pressure":"1012","light":"33"}
Published topic 'Livingroom/voltage with payload 2.62
Published topic 'Livingroom/temperature with payload 75
Published topic 'Livingroom/humidity with payload 48
Published topic 'Livingroom/pressure with payload 1012
Published topic 'Livingroom/light with payload 33
Published topic 'DeviceStatus with payload {"location":"Livingroom","rssi":"-66","mode":"0","ip":"6","channel":"7","sleeptime":"10","uptime":"140"}
Published topic 'Livingroom/rssi with payload -66
Published topic 'Livingroom/mode with payload 0
Published topic 'Livingroom/ip with payload 6
Published topic 'Livingroom/channel with payload 7
Published topic 'Livingroom/sleeptime with payload 10
Published topic 'Livingroom/uptime with payload 140
Published topic 'sensorValues with payload {"location":"Kitchen","voltage":"2.62","temperature":"78","humidity":"49","pressure":"1012","light":"56"}
Published topic 'Kitchen/voltage with payload 2.62
Published topic 'Kitchen/temperature with payload 78
Published topic 'Kitchen/humidity with payload 49
Published topic 'Kitchen/pressure with payload 1012
Published topic 'Kitchen/light with payload 56
Published topic 'DeviceStatus with payload {"location":"Kitchen","rssi":"-42","mode":"0","ip":"16","channel":"7","sleeptime":"5","uptime":"154"}
Published topic 'Kitchen/rssi with payload -42
Published topic 'Kitchen/mode with payload 0
Published topic 'Kitchen/ip with payload 16
Published topic 'Kitchen/channel with payload 7
Published topic 'Kitchen/sleeptime with payload 5
Published topic 'Kitchen/uptime with payload 154

I also wanted to add "Time Received" column to ui-table but do not know how to use node-red time with MM-DD-YY hh-mm-ss format and make it as an index column. In my messages "sleeptime" value is in minutes, "uptime" value is in milliseconds and "ip" value represent last 2 digits of full ip address on local subnet.

Thanks.