Project - deciding when to use arduino code vs nodes

I've been thinking about this for a while as I want to convert my existing project to support MQTT.

Link to my github project
Schematic

In this project there are two inputs from an intruder alarm panel:

  1. Alarm armed or not armed
  2. Sounder on or sounder off

and one output:

  1. Set alarm or unset alarm

So I started looking at converting the code to utilise MQTT so I can create smart actions, i.e. turn lights on when sounder is on.

This lead me to thinking - how much of the processing should be done on the Arduino, vs managing the processing with nodes?

I mainly referring to void checkCombined() function where the processing of the inputs are managed and outputs written to.

Link to code on github

TLDR - advice welcome on when to use code to manage inputs/outputs vs node red logic.

I like to think of the node-red as a central processing unit and the externals as sensors / inputs (Much like all inputs should be fed to a plc and let it do what it does best, processing logic)

If you feed back unfettered inputs you have many opportunities in node-red to condition and combine. If you do logic externally, you loose some capabilities.

There are obviously exceptions...

One such example would be - If you use node-red for monitoring only (i.e. a passive component) then do ALL your logic on the Arduino and only feedback status.

Conversely, if node-red is an integral component (active component) then feed your inputs unprocessed and do all conditioning and logic in node-red.

Ps, these are my usual thoughts when starting most projects.

3 Likes

I have undertaken a similar project recently (and still in the middle of it) - i had a long term heating solution for our house that was originally built using Picaxe Microcontrollers and custom boards, - for various reasons the code to this was lost and never recovered.

I decided to move it across to Arduino and slowly migrated the functionality - initially by grabbing the output that was going to an LCD screen for input to the Arduino, and then slowly impersonating sensors and taking over control of relays etc.

I then in the middle of all of this discovered node-red and decide to move to that as the logic and processing engine, however to make sure i did not get caught in the middle of supporting 3 systems and nothing working, i decided to do it piecemeal.

I initially moved all of the display logic across to Node-red - this gave me some little wins in that the rest of the house could now see what was going on with heating and cooling in the house etc, energy use and the like

I finally finished at the start of this heating season removing the Picaxe from the system, and started to move rudimentary logic and crude control functions into Node-Red and using the Arduino simply to poll sensors and activate relays.

I have also moved a lot of other functions into Node-Red from other systems using Sonoffs and MQTT

Eventually i will probably retire the Arduino and do the lot on Sonoffs, ESP8266 and ESP32 units.

I am just investigating the use of Home-Assistant and whether i want to move to that for more of the Home-Automation and use Node-red for the logic and data aggregation.

Craig

2 Likes

@Steve-Mcl thank you, you explained it much better than I did!

I really like the "passive" vs "active" explanation.

e.g.
-passive would be simply adding some client.publish commands in the code
-active would be reading the digital pin and publishing that state only to the broker.

I'll have a look at how complex the flow would need to be to process the inputs/output. A fair chunk of my code is debouncing the inputs, I could eliminate all of that in one go with RBE nodes I think.

:+1:

Hi @craigcurtin thanks for the detailed reply.

It's great to read your post as I feel it's very familiar. Although it would be great to start with one final system, many people must find it evolves like this. I think it's too much to process in one go, meaning the gradual transition is required.

Same here, I am ending up with ESP8266's (Wemos D1 mini's) and Sonoff Basics with Tasmota. Just got a couple of ESP32's for the more demanding projects.

Also I keep looking at Home-Assistant and am not sure in which direction to go next, I'm using Blynk at the moment which is quite good, and a new "2.0" is imminent.

Yep there is only so much time to put towards something like this and you want to get as many wins as possilbe as you go,

Thats what has me looking at Home-Assistant - i have been tinkering with it for the last 12 months or so (mainly playing around with a docker image) - some things it does are much easier (and a wealth of knowledge and resources to draw on) other things are infuriating. Overall i think using it for the display front end for the home automation system, whilst using Node-Red for the rules, processing and logic seems to be the way to go.

For instance i want to put temperature sensors in each room around the house - no nice way to do that with Node-Red and DS18b20 (with a decent WAF) - but the Xiaomi Weather pucks looks great, are dirt cheap and already have a code hack in Home Assistant to take their output protocol and decode it and present the info on the dashboard (or push it out to NR - which is what i intend to do)

Craig

877,

The process view specified in the old-school, 4+1 Architecture model, might help in your decision making. "The process view deals with the dynamic aspects of the system, explains the system processes and how they communicate, and focuses on the runtime behavior of the system." This is exactly where you are.

Here is an example. Your process view would show Node-RED connecting to an Arduino likely using MQTT over WiFi. An important aspect of this process view isn't that Node-RED is at one end of a connector and the Arduino is at the other. It is the connector itself that impacts your design decision.

What if the Arduino WiFi connection drops at the very second that an event of interest happens? Last Wednesday, I turned on the sprinklers and left for dinner. I returned 4 hours later to find 2" of water in the backyard. The NodeMCU never received the Off event via MQTT. The design implication is that the NodeMCU should do more than receive on and off requests from Node-RED. Maybe it should be that the node receives On and duration. When it completes the task, it publishes state change back to Node-RED to update the UI. In this design, if the endpoint is not able to publish an event back to Node-RED, the worst that happens is that the water turns off in the time expected while the UI behind Node-RED shows that the water is still on.

I have found that it is these kinds of design considerations that are coming to the top for me. And even more so as I rely upon end points that, if not part of a durable design, will cost me significant inconvenience and maybe even real money.

Hope this helps.

2 Likes

Hi @athome

That is an excellent point you make. In the event of a WiFi outage or MQTT broker going down, what I need to happen locally on the esp8266 is quite important. Especially for an intruder alarm.

Thinking back when I designed the project, I made it so if the esp dies and the pin output therefore goes low, it would be made so it arms the alarm panel. This way worst case the alarm is set, much better than it unsetting by mistake.

Lots to consider :+1:

877,

As I was thinking about this, yours is an example that could cost real money if the end point goes rogue since it is common that local police charge a nuisance fine for alarms that result in a police visit or neighbor complaint.

I will share with you that I struggled when I deployed my irrigation controller last month because the NodeMCU-ESP32s would not stay connected. Turns out it takes a wifi-connected esp32 to illuminate what is really going on with the wifi network.

For those who want to do esp32/MQTT/wifi, I highly recommend the EspMQTTClient library, which wraps knolleary's PubSubClient for the specific purpose of adding reconnect (both wifi and mqtt) capability. Without reconnect logic, the endpoint is on its own.

Indeed, as I sat down to write this, I received a reconnect notice from the irrigation controller.

2019.6.28 5:43:35: {"mode":"diagnostic","event":"reconnect","sys":"esp32-irr","topic":"irrigation/monitor","action":"notify","version":"2.0.1","ssid":"Hairy Feet","mac":"pending","mclient":"esp32-irr","free":272936,"invalid":0,"connect":2}

Because of that darn MQTT/Wifi connector, I have found it very useful to implement a ping pub/sub pair to be able to validate that the NodeMCU is connected. When the NodeMCU responds, it also reports free heap as a way to monitor for memory leaks.

I have a local fork of the EspMQTTClient that has an minor tweak (added mqttClient.mqttConnectionCount()) to be able to report startup vs reconnect.

// onConnectionEstablished() is called just once after both 
// the WiFi client and the MQTT are successfully connected
//
// it is in this function that the subscriber is initialized
//
// when the subscriber is initialized, it is possible to 
// assign a function to process the topic payload
//
// NOTE : If this function is not implemented, then no subscriber
//        will be initialized, which means that this sketch will NOT
//        process incoming MQTT messages
void onConnectionEstablished()
{   
  char    msgBuffer[256];
  String  mqttResponse;
    
  // Subscribe to topic and provide the function that will
  // process the payload
  mqttClient.subscribe(TOPIC_EVENT, [](const String & payload) 
  {
      topicPayloadProcess(payload);
  });

  // if connection count is > 1, then the call here is
  // a reconnect
  if(mqttClient.mqttConnectionCount() > 1)
  {
    sprintf(msgBuffer,
            "{\"mode\":\"diagnostic\",\"event\":\"reconnect\",\"sys\":\"%s\",\"topic\":\"%s\",\"action\":\"notify\",\"version\":\"%s\",\"ssid\":\"%s\",\"mac\":\"%s\",\"mclient\":\"%s\",\"free\":%d,\"invalid\":%d,\"connect\":%d}",
            NODE_MCU_NAME,
            TOPIC_MONITOR,
            VERSION,
            WIFI_AP_SSID,
            "pending",
            NODE_MCU_NAME,
            ESP.getFreeHeap(),
            heap_caps_get_free_size(MALLOC_CAP_INVALID),
            mqttClient.mqttConnectionCount());
  }
  else
  {
    sprintf(msgBuffer,
            "{\"mode\":\"diagnostic\",\"event\":\"startup\",\"sys\":\"%s\",\"topic\":\"%s\",\"action\":\"notify\",\"version\":\"%s\",\"ssid\":\"%s\",\"mac\":\"%s\",\"mclient\":\"%s\",\"free\":%d,\"invalid\":%d}",
            NODE_MCU_NAME,
            TOPIC_MONITOR,
            VERSION,
            WIFI_AP_SSID,
            "pending",
            NODE_MCU_NAME,
            ESP.getFreeHeap(),
            heap_caps_get_free_size(MALLOC_CAP_INVALID));
  }

  // Publish the message
  mqttResponse = msgBuffer;      
  mqttClient.publish(TOPIC_MONITOR,
                     mqttResponse);
}

The ping response also reports the reconnect count.

    if(strcmp(eventStr,EVENT_PING) == 0)
    {    
      sprintf(msgBuffer,
              "{\"mode\":\"diagnostic\",\"event\":\"%s\",\"sys\":\"%s\",\"topic\":\"%s\",\"action\":\"notify\",\"msgID\":\"%s\",\"version\":\"%s\",\"ssid\":\"%s\",\"mac\":\"%s\",\"mclient\":\"%s\",\"free\":%d,\"invalid\":%d,\"connect\":%d}",
              EVENT_PING,
              NODE_MCU_NAME,
              TOPIC_MONITOR,
              msgIdStr,
              VERSION,
              WIFI_AP_SSID,
              "pending",
              NODE_MCU_NAME,
              ESP.getFreeHeap(),
              heap_caps_get_free_size(MALLOC_CAP_INVALID),
              mqttClient.mqttConnectionCount());
                          
      mqttResponse = msgBuffer;
      mqttClient.publish(TOPIC_MONITOR,
                         mqttResponse);
1 Like

@athome thank you for the detailed reply!

Yes you are correct it could be costly, I think in my case I am safe as I have just piggy backed onto the alarm panels outputs, and there is a terminal for the set/unset which I am using. But best to be safe!

Thanks for the esp32 info, I did not realise it does not have it's own reconnect logic 'built in'. I'm sure it will come in handy when I venture into trying my recently purchased esp's.

Regarding the ping pub/sub pair, are you using it effectively as a hearbeat? It remind me of a recent post over on the Blynk forum where I asked for some node red tips, it might be of some interest to you : LINK

You might want to have a look at Peter Scargills blog - he has an excellent system whereby timers/watchdogs are set at each ESP - if they do not communicate with the central Pi and reset the watchdog - they will automatically reboot themselves - it is all part of the download and firmare/ecosystem that he has developed

Craig

1 Like

I think indirectly I was advised by someone who might have read that post, as I have the exact same setup already running:

Also I purchased a watchdog timer from Superhouse TV (youtube) which looks great, while I was waiting I made my own using a TPL5010 after some advice on the Arduino forums - LINK.

I have had both working with simple code, but I've yet to implement one into a project :blush: