Mqtt like link node based on topics

General
42

for example in case of event handler.

Let's say I have 100 subflow which handle 100 physical devices via zigbee.

The user would like to create a light control. If the ambient light less than 10 lux (light sensor #12) the garden light (lamp #13) should turn on.

In node red the light sensor subflow send status change (‘topic link out’ node, topic name: /status/sensor12, payload: “100 lux”)

The user set in the UI of event handler that he wants to use the sensor12 for light sensing, and set the level :100 lux, so the event handler have to set the topic of its ‘topic link in’: /status/sensor12

But two days later, the user thinks that sensor 13, not sensor 12 should be the source for turning on the garden light, so after that ‘topic link in’ node of event handler should pay attention to sensor 13 (/status/sensor13).

Certainly, the event handler can listen all status topic: /status/#, but in this case it gets statuses of all devices (100)

43

I think that this is a very specific use case, so I don't think that it is worth to have a own node for it.
Why can the user not just change it in Node-RED itself?

You can use something like (untested)

[
    {
        "id": "6b46bef0.ac7d8",
        "type": "function",
        "z": "da6976fb.f727d",
        "name": "",
        "func": "if(msg.setTopic === true) {\n    context.set('topic', msg.topic);\n} else {\n    var topic = context.get('topic');\n    if(topic === msg.topic) {\n        return msg;\n    }\n}\n",
        "outputs": 1,
        "noerr": 0,
        "x": 670,
        "y": 440,
        "wires": [
            []
        ]
    },
    {
        "id": "91cf94c.15fa468",
        "type": "topic-link in",
        "z": "da6976fb.f727d",
        "topic": "status/#",
        "appendTopic": true,
        "name": "",
        "x": 470,
        "y": 440,
        "wires": [
            [
                "6b46bef0.ac7d8"
            ]
        ]
    },
    {
        "id": "3d4fc809.8fd8d8",
        "type": "inject",
        "z": "da6976fb.f727d",
        "name": "",
        "topic": "status/sensor13",
        "payload": "",
        "payloadType": "str",
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "x": 280,
        "y": 500,
        "wires": [
            [
                "1b6a6a41.3d4f76"
            ]
        ]
    },
    {
        "id": "1b6a6a41.3d4f76",
        "type": "change",
        "z": "da6976fb.f727d",
        "name": "",
        "rules": [
            {
                "t": "set",
                "p": "setTopic",
                "pt": "msg",
                "to": "true",
                "tot": "bool"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 490,
        "y": 500,
        "wires": [
            [
                "6b46bef0.ac7d8"
            ]
        ]
    }
]

If you put it into a subflow, it is reusable

44

Going back to whether to use MQTT or not, really don't worry about the load on the server (use Mosquitto). Unless you are publishing thousands of messages a second then a Pi 4 is not going to even notice it.
When developing software don't try and optimise it before you know there is a problem. You will waste time optimising it and I can guarantee that the bottleneck will never be where you think it is. Always build it the easiest way the first time and optimise it later if you need to (which you almost certainly won't need to do). Concentrate on the important things for maintainability, such as separating the Input flows from the system logic and from the flows that drive the UI. That will make it easier later on when you have to make changes to the system.

5 Likes
45

Agree that this seems to specific for a custom node. But I don't agree that you should change a flow for this. Instead put the configuration into a data structure of some sort so that you can easily create a UI to change the assignments.

1 Like
46

The users (my family – my wife and my children) can’t modify node-red. They don’ know what is node red. They only use UI (Dashboard) and they select ’garden light meter' (node red id: light sensor 12) from a dashboard list.

The idea is to have an absolute generic dashboard which displays all devices publishing in topic link, but you never know how many or which devices will be there. That´s why anyone needs to include the subscribtion inside of a flow and using the data created before to subscribe to the right device and use it´s data in the following nodes.
Well, sometimes, depending on user intervention on the Dashboard, it may need to change the topic of 'topic link in' node for dashboard event management.

Recap: if you use on dashboard a dynamic list (for example device list, error log level list, etc), you will have to change the topic name of ‘topic link in’ dynamically.
I think you are right, no need to create a new node, only the existing ‘topic link in’ node should be supplemented with an input.
So, I think the dynamic ‘topic link in’ will be very useful someone else too who use dropdown list on dashboard.

47

Really wondering if your thoughts & plans will find it's way to success. Isn't the ultimate goal to make a home automation system that is so simple to use, it should almost not be noticed by anyone? Just doing it's stuff in the background fully automatic, all depending on your rules, no need for user interaction at all, a bit of AI added on top, using cameras etc

I guess you are targeting a high WAF score (Wife Acceptance Factor)?

Selecting 'light meters' sounds to me, too complicated. Such input data I would use as input to a rule engine & AI, all in background, fully hidden to end users

The more you can automize, the better. At least that would be key to me. In normal operation, my own home automation system is not requiring any user intervention at all, it has all those parts you may think of. In addition I have spent a massive effort in making it "self-repairing" if parts would need to restart for some various reasons (this will likely happen)

But now I'm drifting off-topic

48

The subscription would be lost if you redeploy/restart Node-RED. Not sure if that is what you want.
I posted a solution for you problem, which I think is quite adequate, since with this solution you can also solve the problem of persisting the topics etc. or do what Julian suggests and have that coming from a configuration datastructure etc.

49

I’d like to make my smart home system as follows:

The node red system would basically be divided into 7 physical subflow parts:

I. database (sqlite or mysql) management:

  • current / daily / annual status of the device (status, event, action)
  • device configuration
  • error, warning

II. system startup (rpi start, deploy)

  • Load devices configuration data and devices last status from database. Setting the state (according to what is loaded from the configuration database) on the output devices (previous state, specified state, state in progress at shutdown, etc.…)
  • here I would set the dynamic ‘topic link in’ topic name (from the configuration database) which node is in the event handler subflow).

III. power failure management

In the event of a power failure (certainly rpi is on UPS, but it monitors the power failure), the process is like the previous point for each output device depending on the configuration database (eg. the garage door is closing when a power failure happens. After switching the power back on, based on the configuration database, either the closing will continue (closing command), or it will not do anything or it will is go to a basic state (open / close)

IV. Voice Control

  • Amazon Alexa
  • Siri
  • Android

V. Notification

  • E-mail
  • mobile push message, etc ..
  • etc ...

VI. basic relationships between devices

Subflows manages the basic relationship between different devices: e.g .:

  • bedroom switch pressed: bedroom light on
  • bedroom switch double pressing: bedroom lamp brightness set to 50%, etc.….
  • Someone opened the garage door: message on the mobile that the garage is open
  • etc ..

The subflows above would be handled in the background by node-red. If I like to add a new device (sensor, actuator (eg relay), garage door motor, etc.) to the smart home system, I would add it in the node-red editor with the necessary extra by inserting and modifying subflows, databases, other node red elements

The next schedule / event management subflow would be what the user (wife) could set up in the dashboard:

VII. User automation (schedule / event) management

These automations would be what the user (my wife / kids) could set up on the Dashboard. The user can add a new event to the events list. They choose which change of sensor state will be listen (bedroom temperature sensor), which actuator element should respond (IR remote for air conditioning unit). In addition they can scheduling this event. The selected sensor (eg temperature sensor) and actuator element (eg IR remote) would be stored in the event database, the scheduling (eg from July to September) would be stored in the schedule database. These data would be loaded from the event and schedule database when rpi is (re)started or Node Red deployed and set the topic of 'topic link in' node of event manager dynamically, that the event handler subflow receives only the status messages which the user want to monitor (changing of bedroom temperature sensor), and it doesn’t receive all status messages of all devices.

As I wrote, and you wrote it clearly too (with example), I can do it without dynamic topic addressing, but in this case the event handler ‘topic link in’ node will get a message status changing of all device, instead of just get one message when bedroom temperature sensor send status changing message.

That’s why I would like a dynamic ‘topic link in’ node.

I still don't understand why you say this is so specific, since all smart home apps handle user-configurable event handling (e.g. Xiaomi home app (Device selection -> Automation -> 'if' and 'then' options) (eWeLink app: Device selection -> Auto -> 'Control by' and 'set' option).

In these app the user can set on the user interface what should happen if a sensor changes its status The devices must be managed dynamically from the perspective of the event handler.

Certainly, I'm glad you did the 'topic link in' and 'topic link out' nodes, but I think this message transfer among nodes would be complete with dynamic 'topic link in'. I think many other places might need it, probably that's why somebody did the mqtt dynamic topic input node addition as well.

Thanks again to both of you for devoting so much time to me.

If you do not add 'topic link in' to the topic option via message, I will stick to mqtt messaging, as it is logically a better solution for me than the input node receives all device messages and the subsequent function node should select the one message is required from each message, or I try to learn the forts of custom node creation and supplement the ‘topic link in’ node with an input.

Thanks again for the help.

50

In my opinion, configuring that is what Node-RED is for and not an app built on top of it. The other apps you are mentioning aren't like Node-RED. And I also believe that your wife and your children could change that in Node-RED itself.

Anyway, here you go GitHub - cinhcet/node-red-contrib-topic-link at dynamicInNode
I don't like the additional input on the node. Therefore, I am undecided yet whether I will publish that on npm.
So you need to clone that repository, e.g. into ~/.node-red/node-red-contrib-topic-link, then git checkout dynamicInNode in the folder and then install via npm install node-red-contrib-topic-link from ~/.node-red. Remember to uninstall the other first.

I have quickly programmed that without any testing, so no guarantee that it will work. The topic should be set via the topic property (what else :-))

Edit: If anyone else has opinions, let us know :slight_smile:

51

You could make the input optional. Hide it by default and put a checkbox somewhere in the nodes config to make it appear/enable it. Could be an option to have both.

Johannes

52

Or alternatively just:

npm install cinhcet/node-red-contrib-topic-link#dynamicInNode

No need to uninstall first IIRC. :slightly_smiling_face:

1 Like
53

Thank you very much for your help, but it doesn't work.

After deploying I inject topic (/status) to 'topic link in', then I inject a message to 'topic link out'. The topic of message: /status too, but the message doesn't appear at the output of 'topic link in'

54

ok, I tested it

[{"id":"cc2ca139.088258","type":"topic-link in","z":"65ad4c53.2cfdd4","topic":"","appendTopic":true,"name":"","x":790,"y":620,"wires":[["2e7838c.0506148"]]},{"id":"58219ccb.75dd5c","type":"topic-link out","z":"65ad4c53.2cfdd4","topic":"topic/test","name":"","x":390,"y":620,"wires":[]},{"id":"4160fc49.7f339c","type":"inject","z":"65ad4c53.2cfdd4","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"payload":"","payloadType":"date","x":210,"y":620,"wires":[["58219ccb.75dd5c"]]},{"id":"2e7838c.0506148","type":"debug","z":"65ad4c53.2cfdd4","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":970,"y":620,"wires":[]},{"id":"3edb7ed6.c4bd6a","type":"inject","z":"65ad4c53.2cfdd4","name":"","props":[{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"topic/test","x":620,"y":620,"wires":[["cc2ca139.088258"]]}]

and it works?

This was made in the current beta... Probably the inject node will not work in 1.0

Post your flow. Saying "it doesn't work" is always problematic without providing more information

55

I see what you are saying. You are also using dynamic topic out. Will push a fix.
If you set the topic in the out node, it should work.

I like it less and less. Now you can misuse that node to dynamically send messages around to everywhere :stuck_out_tongue:

56

fixed.

But please, think twice whether this is a good approach of having both dynamic topic in and out.

And MQTT topics don't have to start with a /

57

Thank you very much again.

58

does it work?

Your welcome :slight_smile:

59

It works great.

60

I'll just reiterate what others have said about dynamic topic names. This should not be needed if you have chosen your topic layout correctly. MQTT allows for wildcards so choosing a good topic layout such as the Homie convention will eliminate most issues. However, it is not uncommon to end up with several topic structures in MQTT for different purposes - simply to make life easier.

For example, I'm slowly adopting the Homie convention for all of my detailed device data but that still means I need to make sure that all of my devices publish things like sensor data and switch status in a common format so that I can subscribe to something like DEVICES/#/sensors/temperature and get all temperature readings fed to a Node-RED flow no matter how many devices/sensors I have.

I don't need to know what device names there are and I don't need to dynamically update the subscription.

However, I will, of course, generally need a bunch of other data to be able to make up a sensible UI. While I can do that using MQTT - DEVICES/# for example and then sort it all out in Node-RED, it is much easier to keep that data in structured format in a persistent Node-RED global variable even though I may well still push a lot of that "meta-data" to MQTT as well. Things like room names and mappings of devices to rooms. I'll update these from a UI, saved in Node-RED and pushed to MQTT.

It is also not at all uncommon to create specific topic structures to make life easier. For example, I often use something like TEMPERATURES/<deviceid> or ROOMS/<room name>/temperature etc as convenience topics. But I keep the raw data under the DEVICES topic as you never know what you might need at some other point. Pushing data to these topics is simple with Node-RED and wildcard subscriptions.

1 Like
61

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.