Trying to work out why not getting updates for a particular output from a script

[dclear]

I have a PHP script which returns states for the extensions on my home's phone network. I recently moved from the exec node to the daemon node which bizarrely necessitated a couple of unwanted changes to the way my flow worked. I had to introduce two function nodes to be able to split the messages from the script in a way that hadn't been necessary before. Anyway a knock on effect that I've only just noticed is that I'm no longer getting any updates for the '4004' extension. Don't know why. I've moved the telegram node, which I'm using a debug node to the output of the script (before I had it after the second function node) and I can see they're being sent, it's just my regex matching isn't working anymore for some reason. Any help would be much appreciated! Thank you

[{"id":"3aedb22.998dc4e","type":"switch","z":"36ca7898.7bdf48","name":"","property":"payload","propertyType":"msg","rules":[{"t":"regex","v":"4001","vt":"num","case":true},{"t":"regex","v":"4003","vt":"num","case":true},{"t":"regex","v":"4004","vt":"num","case":true}],"checkall":"true","repair":false,"outputs":3,"x":510,"y":200,"wires":[["a9c44c6a.9af79"],["cb49d4e.b009928"],["59d0b5f9.e69c0c"]]},{"id":"a9c44c6a.9af79","type":"switch","z":"36ca7898.7bdf48","name":"","property":"payload","propertyType":"msg","rules":[{"t":"regex","v":"idle","vt":"str","case":true},{"t":"regex","v":"ringing","vt":"str","case":true},{"t":"regex","v":"inuse","vt":"str","case":true}],"checkall":"true","repair":false,"outputs":3,"x":650,"y":100,"wires":[["addaf9f1.d01cc8"],["70236d08.e292e4"],["5d383102.265e"]]},{"id":"cb49d4e.b009928","type":"switch","z":"36ca7898.7bdf48","name":"","property":"payload","propertyType":"msg","rules":[{"t":"regex","v":"idle","vt":"str","case":true},{"t":"regex","v":"ringing","vt":"str","case":true},{"t":"regex","v":"inuse","vt":"str","case":true}],"checkall":"true","repair":false,"outputs":3,"x":650,"y":200,"wires":[["9871d79b.d57688"],["78a437d0.dc7c88"],["519cfd63.413db4"]]},{"id":"59d0b5f9.e69c0c","type":"switch","z":"36ca7898.7bdf48","name":"","property":"payload","propertyType":"msg","rules":[{"t":"regex","v":"idle","vt":"str","case":true},{"t":"regex","v":"ringing","vt":"str","case":true},{"t":"regex","v":"inuse","vt":"str","case":true}],"checkall":"true","repair":false,"outputs":3,"x":650,"y":300,"wires":[["bc754b90.818858"],["19885700.fabc69"],["a41a4204.29fe9"]]},{"id":"addaf9f1.d01cc8","type":"change","z":"36ca7898.7bdf48","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"IDLE","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":820,"y":40,"wires":[["8ff384c.bfe6778"]]},{"id":"70236d08.e292e4","type":"change","z":"36ca7898.7bdf48","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"RINGING","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":820,"y":80,"wires":[["8ff384c.bfe6778"]]},{"id":"5d383102.265e","type":"change","z":"36ca7898.7bdf48","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"IN_USE","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":820,"y":120,"wires":[["8ff384c.bfe6778"]]},{"id":"8ff384c.bfe6778","type":"mqtt out","z":"36ca7898.7bdf48","name":"","topic":"exten/4001","qos":"","retain":"","broker":"459c73d.68a4d8c","x":1020,"y":80,"wires":[]},{"id":"9871d79b.d57688","type":"change","z":"36ca7898.7bdf48","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"IDLE","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":820,"y":160,"wires":[["25f7721a.7960ce"]]},{"id":"78a437d0.dc7c88","type":"change","z":"36ca7898.7bdf48","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"RINGING","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":820,"y":200,"wires":[["25f7721a.7960ce"]]},{"id":"519cfd63.413db4","type":"change","z":"36ca7898.7bdf48","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"IN_USE","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":820,"y":240,"wires":[["25f7721a.7960ce"]]},{"id":"25f7721a.7960ce","type":"mqtt out","z":"36ca7898.7bdf48","name":"","topic":"exten/4003","qos":"","retain":"","broker":"459c73d.68a4d8c","x":1020,"y":200,"wires":[]},{"id":"bc754b90.818858","type":"change","z":"36ca7898.7bdf48","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"IDLE","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":820,"y":280,"wires":[["152c5552.67110b"]]},{"id":"19885700.fabc69","type":"change","z":"36ca7898.7bdf48","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"RINGING","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":820,"y":320,"wires":[["152c5552.67110b"]]},{"id":"a41a4204.29fe9","type":"change","z":"36ca7898.7bdf48","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"IN_USE","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":820,"y":360,"wires":[["152c5552.67110b"]]},{"id":"152c5552.67110b","type":"mqtt out","z":"36ca7898.7bdf48","name":"","topic":"exten/4004","qos":"","retain":"","broker":"459c73d.68a4d8c","x":1020,"y":320,"wires":[]},{"id":"fc3d0ba.7bad3f8","type":"daemon","z":"36ca7898.7bdf48","name":"","command":"/usr/bin/php","args":"/root/lgaetz-hinterland.php","autorun":true,"cr":false,"redo":true,"op":"string","closer":"SIGKILL","x":250,"y":100,"wires":[["d0102568.da0828","d1933095.e0025"],[],[]]},{"id":"d0102568.da0828","type":"function","z":"36ca7898.7bdf48","name":"","func":"var values = msg.payload.trim().split('\\t');    // trim off the crlf first then split into parts   \nmsg.payload = values[0];  // replace payload with the reading we want\nreturn msg;","outputs":1,"noerr":0,"x":230,"y":260,"wires":[["43c98d86.65d534"]]},{"id":"43c98d86.65d534","type":"function","z":"36ca7898.7bdf48","name":"","func":"if (msg.payload.match(/\\r/g)) {\n    const msgArr = msg.payload.split('\\r');\n    msgArr.forEach(item => node.send({payload:item}));\n    return;\n} else {\n    return msg;\n}","outputs":1,"noerr":0,"x":380,"y":300,"wires":[["3aedb22.998dc4e"]]},{"id":"d1933095.e0025","type":"telegrambot-notify","z":"36ca7898.7bdf48","name":"","bot":"2fe21bf4.832cd4","chatId":"763138506","message":"","parseMode":"","x":460,"y":100,"wires":[]},{"id":"459c73d.68a4d8c","type":"mqtt-broker","z":"","name":"","broker":"192.168.0.10","port":"1883","clientid":"node-red","usetls":false,"compatmode":true,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"","willQos":"0","willPayload":""},{"id":"2fe21bf4.832cd4","type":"telegrambot-config","z":"","botname":"pbxbot","usernames":"","chatIds":"---","pollInterval":"300"}]`

[Paul-Reed]

Please see this post - How to share code or flow json
and edit you post above (pencil icon).

Presently, the flow cannot be be loaded, because Discourse makes changes to it.

[dclear]

Thanks Paul - done - interestingly I did again as I had before select the code and click the preformatted text button, but this did not:
This will automatically insert three back-tick characters - ````` - on their own line before and after the code.
instead it just inserted one ` at the beginning and one at the end.
I had to manually do a new line and type out two more characters in order for it to display on one line only.

[Colin]

It works more reliably if you click the button then paste the text in.

[dclear]

I think this shows what's happening - occasionally a message ISN'T being split and is causing the wrong messages to be sent for the wrong situations.

Could I simply split after the first "status: $" somehow?

[dclear]

I've had another pop at trying to do things differently (doing some string cleanup before feeding to the switch node etc) and STILL for some reason messages aren't reliably being routed to the right nodes. I know not why. I've looked at the various debug nodes and the messages come in from the PHP script just fine but they don't reliably get to the switch node.

[{"id":"3aedb22.998dc4e","type":"switch","z":"36ca7898.7bdf48","name":"","property":"payload","propertyType":"msg","rules":[{"t":"cont","v":"4001","vt":"str"},{"t":"cont","v":"4003","vt":"str"},{"t":"cont","v":"4004","vt":"str"}],"checkall":"true","repair":false,"outputs":3,"x":510,"y":200,"wires":[["a9c44c6a.9af79","94081938.f32de8"],["a481224d.1b7db","94081938.f32de8"],["914994a5.ab24c8","94081938.f32de8"]]},{"id":"a9c44c6a.9af79","type":"switch","z":"36ca7898.7bdf48","name":"","property":"payload","propertyType":"msg","rules":[{"t":"cont","v":"status: 0","vt":"str"},{"t":"cont","v":"status: 8","vt":"str"},{"t":"cont","v":"status: 1","vt":"str"}],"checkall":"true","repair":false,"outputs":3,"x":650,"y":100,"wires":[["addaf9f1.d01cc8"],["70236d08.e292e4"],["5d383102.265e"]]},{"id":"addaf9f1.d01cc8","type":"change","z":"36ca7898.7bdf48","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"IDLE","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":820,"y":40,"wires":[["8ff384c.bfe6778"]]},{"id":"70236d08.e292e4","type":"change","z":"36ca7898.7bdf48","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"RINGING","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":820,"y":80,"wires":[["8ff384c.bfe6778"]]},{"id":"5d383102.265e","type":"change","z":"36ca7898.7bdf48","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"IN_USE","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":820,"y":120,"wires":[["8ff384c.bfe6778"]]},{"id":"8ff384c.bfe6778","type":"mqtt out","z":"36ca7898.7bdf48","name":"","topic":"exten/4001","qos":"","retain":"","broker":"459c73d.68a4d8c","x":1020,"y":80,"wires":[]},{"id":"9871d79b.d57688","type":"change","z":"36ca7898.7bdf48","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"IDLE","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":820,"y":160,"wires":[["25f7721a.7960ce"]]},{"id":"78a437d0.dc7c88","type":"change","z":"36ca7898.7bdf48","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"RINGING","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":820,"y":200,"wires":[["25f7721a.7960ce"]]},{"id":"519cfd63.413db4","type":"change","z":"36ca7898.7bdf48","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"IN_USE","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":820,"y":240,"wires":[["25f7721a.7960ce"]]},{"id":"25f7721a.7960ce","type":"mqtt out","z":"36ca7898.7bdf48","name":"","topic":"exten/4003","qos":"","retain":"","broker":"459c73d.68a4d8c","x":1020,"y":200,"wires":[]},{"id":"bc754b90.818858","type":"change","z":"36ca7898.7bdf48","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"IDLE","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":820,"y":280,"wires":[["152c5552.67110b"]]},{"id":"19885700.fabc69","type":"change","z":"36ca7898.7bdf48","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"RINGING","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":820,"y":320,"wires":[["152c5552.67110b"]]},{"id":"a41a4204.29fe9","type":"change","z":"36ca7898.7bdf48","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"IN_USE","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":820,"y":360,"wires":[["152c5552.67110b"]]},{"id":"152c5552.67110b","type":"mqtt out","z":"36ca7898.7bdf48","name":"","topic":"exten/4004","qos":"","retain":"","broker":"459c73d.68a4d8c","x":1020,"y":320,"wires":[]},{"id":"fc3d0ba.7bad3f8","type":"daemon","z":"36ca7898.7bdf48","name":"","command":"/usr/bin/php","args":"/root/lgaetz-hinterland.php","autorun":true,"cr":false,"redo":true,"op":"string","closer":"SIGKILL","x":170,"y":100,"wires":[["4b63f892.ef4dd8","dd197c66.3a727"],[],[]]},{"id":"7903f5b.e94f50c","type":"string","z":"36ca7898.7bdf48","name":"","methods":[{"name":"between","params":[{"type":"str","value":"exten:  "},{"type":"str","value":" statustext"}]}],"prop":"payload","propout":"payload","object":"msg","objectout":"msg","x":450,"y":60,"wires":[["3aedb22.998dc4e"]]},{"id":"a481224d.1b7db","type":"switch","z":"36ca7898.7bdf48","name":"","property":"payload","propertyType":"msg","rules":[{"t":"cont","v":"status: 0","vt":"str"},{"t":"cont","v":"status: 8","vt":"str"},{"t":"cont","v":"status: 1","vt":"str"}],"checkall":"true","repair":false,"outputs":3,"x":650,"y":200,"wires":[["9871d79b.d57688"],["78a437d0.dc7c88"],["519cfd63.413db4"]]},{"id":"914994a5.ab24c8","type":"switch","z":"36ca7898.7bdf48","name":"","property":"payload","propertyType":"msg","rules":[{"t":"cont","v":"status: 0","vt":"str"},{"t":"cont","v":"status: 8","vt":"str"},{"t":"cont","v":"status: 1","vt":"str"}],"checkall":"true","repair":false,"outputs":3,"x":650,"y":320,"wires":[["bc754b90.818858"],["19885700.fabc69"],["a41a4204.29fe9"]]},{"id":"4b63f892.ef4dd8","type":"function","z":"36ca7898.7bdf48","name":"","func":"var values = msg.payload.trim().split('\\t');    // trim off the crlf first then split into parts   \nmsg.payload = values[0];  // replace payload with the reading we want\nreturn msg;","outputs":1,"noerr":0,"x":330,"y":120,"wires":[["7903f5b.e94f50c","d1be6587.ab6b18"]]},{"id":"dd197c66.3a727","type":"debug","z":"36ca7898.7bdf48","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":260,"y":280,"wires":[]},{"id":"d1be6587.ab6b18","type":"debug","z":"36ca7898.7bdf48","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":350,"y":220,"wires":[]},{"id":"94081938.f32de8","type":"debug","z":"36ca7898.7bdf48","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":630,"y":260,"wires":[]},{"id":"459c73d.68a4d8c","type":"mqtt-broker","z":"","name":"","broker":"192.168.0.110","port":"1883","clientid":"node-red","usetls":false,"compatmode":true,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"","willQos":"0","willPayload":""}]

[Colin]

First work out, using debug nodes, which node is failing. Then post the debug showing what is going into that node, show us the config of the node and tell us what is or is not happening. We don't need the whole flow and lots of debug, just this information.

[Steve-Mcl]

Hi,

In an attempt to help - I have a few questions...

• Are you in control of the values returned from the PHP node? (i.e. if it greatly simplified things in the flow, could/would you modify the PHP output if needed?)

• Can you show us what comes out of the PHP node? (use the copy button on the debug message and paste it as a code block - that way someone attempting to help wont have to re-type it from a screen shot)

• Can you explain how you want that data presented to the MQTT out nodes

[dclear]

The annoying thing is it seems to vary each time which of the nodes do and don't work properly. I have every confidence that the output of the script must vary ever so slightly somehow (?!) because occasionally it works perfectly. Mostly, though, one of the mqtt outputs fails to receive the 'idle' sensor payload but it's not guaranteed to be the same one.

[dclear]

@Steve-Mcl

In an attempt to help - I have a few questions...

Your attempt to help is very welcome! Thank you!

@Steve-Mcl

• Are you in control of the values returned from the PHP node? (i.e. if it greatly simplified things in the flow, could/would you modify the PHP output if needed?)

Yes I could easily modify the PHP output as it sits on my server

@Steve-Mcl

• Can you show us what comes out of the PHP node? (use the copy button on the debug message and paste it as a code block - that way someone attempting to help wont have to re-type it from a screen shot)

Happily!

exten: 4002 status: 4 statustext: Unavailable
exten: 4002 status: 0 statustext: Idle
exten: 4002 status: 8 statustext: Ringing
and how they appear in the debug window:

note the pesky carriage returns...

@Steve-Mcl

• Can you explain how you want that data presented to the MQTT out nodes

Well, basically how I have it in my flow at the moment is exactly how I want it to behave- I'd like to send MQTT messages when one of three internal extensions is in use, ringing and idle

Thanks!

P.S In a way I don't know if I'm answering (or beginning to answer my own question here, but I find it odd the way the name of the property that has been passed in has exten/4001 in it for some reason and the payload has nothing to do with exten/4001...)

[Steve-Mcl]

@dclear

Yes I could easily modify the PHP output as it sits on my server

Then I believe the simplest solution would be to modify the PHP output to return valid JSON.

Then you will be able to access msg.x and msg.y (or whatever you call them.

Example, instead of sending this...

@dclear

exten: 4002 status: 4 statustext: Unavailable
exten: 4002 status: 0 statustext: Idle
exten: 4002 status: 8 statustext: Ringing

{
  "exten_4002_status": 4, 
  "exten_4002_statusText":  "Unavailable"
}

Then pass the value to a JSON node then to a debug node where you will see you have a perfectly formed JS object where you can simply grab any property of the msg (without splitting strings)

While what I chose might not fit you solution perfectly (I don't know what the data means or what you are trying to achieve) the principle remains the same. Send valid json then it's simple to convert to a JS object and simple to grab the values from that JS object.

[dclear]

hmmm. Thank for the advice. I just went back to the exec node and (while it's running) it's UBER reliable and does not suffer from any misfires. There's something wrong with the daemon node and I don't know what it is. So I'm stuck between having the exec node work reliably (but occasionally falls down for reasons unknown, sometimes it takes a few days, sometimes a few hours, and the daemon node never falls down but performs its job super poorly. I don't think editing the PHP file to send proper JSON wouldn't necessarily help as it looks like the daemon node has issues. Do correct me if I'm wrong!

[Steve-Mcl]

My advice would reduce you flow of about 20 nodes to about 3 nodes & thus remove great complexity in breaking up those strings. The best way to pass data to a JS application is via JSON (it's native and dependable). Splitting strings is pretty much the last choice.

I must have missed the bit about unreliable delivery / daemons (apologies)

Whatever works for you.

Ps. What is it you do in PHP that you couldn't do in node-red (i.e. cut the middle man out)?

[dclear]

@Steve-Mcl

Ps. What is it you do in PHP that you couldn't do in node-red (i.e. cut the middle man out)?

I'd LOVE to cut the middleman out...if you think node-red can do this natively I'm all for it!

There is an AMI connector but it's a bit broken because it doesn't filter events (even though it claims it can) and Asterisk generates a LOT of events so I was worried I would be overloading my little Pi running the AMI connector rather than the simple and selective PHP script

[Steve-Mcl]

@dclear

I'd LOVE to cut the middleman out...if you think node-red can do this natively I'm all for it

Node-red was born to do this kinda stuff. It's its bread n butter.

I see nothing insurmountable in there

Create TCP connection.
Send TCP request & parse string result
Send to MQTT

Your pi is enormously capable of doing this task. Arguably (due to node is event loop) it can do it better.

I recommend you give it a go & you will learn loads as you go. Bonus.

If you get stuck, come back and ask specific questions. Folks on here are very helpful when someone tries & gets stuck.

[Colin]

@dclear

The annoying thing is it seems to vary each time which of the nodes do and don't work properly.

If you decide to continue with down this route put debug nodes everywhere significant, give them names to more easily identify them, and capture one of the failures. Then you can analyse that one and see what is going wrong. If necessary set them to log to the console, then they will be written to the log so you can analyse them at your leisure. You can be assured that the Switch node does work, so there must be something wrong with the data or the configuration.

[dclear]

Thanks Colin and Steve-Mcl. Redid using the AMI connector node and now works perfectly. Thank you.

@Colin

and capture one of the failures

I did try that but there was nothing different about it. And my point was kind of proven when I went back to the exec node it began working reliably again. ie my flow was not the problem it was the daemon node. So then I began looking for reliability and accuracy. That led me back to the AMI node. Thanks all.

[Colin]

@dclear

I did try that but there was nothing different about it.

So you have a documented situation where you can provide the details of the message going into a node, the configuration of the node, and details of what comes out and show that it did not respond as it should. In that case you have found a bug.

[dclear]

@Colin

In that case you have found a bug.

Yes, I think so, too. I've opened a ticket.

[Colin]

@dclear

I've opened a ticket

Can you post a link to the ticket please?