Help with logging to a .csv file at a user-alterable time-base

[CADonkey]

Hi,

I am trying to log a set of data (constructed from various sensor inputs over serial and OPCUA links) to a .csv file and struggling how to resolve a few issues that I hope someone can help with.

I want to have the logging enabled/disabled by the user toggling a "LOG" switch. I want to the append a .csv file every x seconds, whereby the x seconds (log-rate) is obtained from a flow-var obtained from a numeric input that the user can vary.

This is what I currently have (image below). This is working OK apart from two issues.

The two issues that I have are....

1. Once the "LOG" switch is set high, this works as intended. However when the "LOG" switch is set low, the .csv is still updated (logging continues indefinitely). In other words, the AND logic doesn't work correctly with the two inputs of the switch and the inject node sending a boolean True as the payload (repeats every 5 seconds).

I have tried using various combinations of triggers and delays, instead of an inject-repeat, but still have the same issue of once the logging is active, it doesn't deactivate once the "LOG" switch is set low.

2. Because I am using an inject node set to govern the log rate, there is only the option of entering a time value for the repeat rate. I would like to be able to set the repeat rate using a flow-var value, so that it can be linked to a user-input.

Help!

[jbudd]

Hi Martin, welcome to the forum.

Frankly, the picture of your flow is of very little use without an export of the code.

You can export your code by selecting Export from the hamburger menu (or CTRL-e).
Make sure that "Current flow" is selected and click "Copy to clipboard"
There is an explanation of how to paste the code in the forum at https://discourse.nodered.org/t/how-to-share-code-or-flow-json/506

That said, here is a snippet showing one way you can "switch off" the flow of sensor data. It uses a non-standard node - node-red-contrib-simple-gate which has to be installed. Also inject nodes to simulate the dashboard switch settings.
It does not address the issue of configurable log frequency, though that is cerainly possible.

Here is the code. Copy from here and import it to Node-red via the hamburger menu.

[{"id":"63e15978c5f11334","type":"inject","z":"1ff7a51a41da7b57","name":"Sensor data (every second)","props":[{"p":"payload"}],"repeat":"1","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":240,"y":100,"wires":[["0426f7dde454aa0d"]]},{"id":"2214edb78e4af385","type":"inject","z":"1ff7a51a41da7b57","name":"Start Log","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"control","payload":"START","payloadType":"str","x":300,"y":180,"wires":[["0426f7dde454aa0d"]]},{"id":"8b1b26305066a13b","type":"inject","z":"1ff7a51a41da7b57","name":"Stop Log","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"control","payload":"STOP","payloadType":"str","x":300,"y":220,"wires":[["0426f7dde454aa0d"]]},{"id":"0426f7dde454aa0d","type":"gate","z":"1ff7a51a41da7b57","name":"","controlTopic":"control","defaultState":"closed","openCmd":"START","closeCmd":"STOP","toggleCmd":"toggle","defaultCmd":"default","statusCmd":"status","persist":false,"storeName":"memory","x":490,"y":180,"wires":[["c1ba9b89973d643b"]]},{"id":"c1ba9b89973d643b","type":"debug","z":"1ff7a51a41da7b57","name":"debug 318","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":670,"y":180,"wires":[]}]

[CADonkey]

Hi J,

Thanks for that, it was really helpful and works OK with what I'm doing

I've exported the front end of the logging section, just up to the point that the formatted CSV row string is then passed on the to the filename constructor, csv node and file append (as the back end works OK).

My second problem is how to use a user-enterable value as the log-rate, via a numeric input (with that value then sent to a flow-var).

I've got it currently setup used an inject node with a 5 second repeat in the example and I've tried it with a trigger and a loop timer (additional node install), but I haven't figured out a way to use a flow-var as part of the timing). Any suggestions?

Many thanks.

[
    {
        "id": "3d7d17a620f1054b",
        "type": "tab",
        "label": "Flow 2",
        "disabled": false,
        "info": "",
        "env": []
    },
    {
        "id": "27f2ba6bc9cd4809",
        "type": "function",
        "z": "3d7d17a620f1054b",
        "name": "Configure CSV row",
        "func": "var PUMP_RPM = flow.get('PUMP_RPM') || 0;\nvar PRESSURE = flow.get('PRESSURE') || 0;\nvar TIMEO = flow.get('TIMEO') || 0;\nvar MASS = flow.get('MASS') || 0;\nvar TEMP = flow.get('TEMP') || 0;\nvar DO = flow.get('DO') || 0;\nvar NEEDLE = flow.get('NEEDLE') || 0;\nvar COMMENT = flow.get('COMMENT') || 0;\nmsg.payload = {\n    \"Timestamp\": TIMEO,\n    \"Pressure_mbar\": PRESSURE,\n    \"Pump_RPM\": PUMP_RPM,\n     \"Mass_gr\": MASS,\n     \"Temp\" : TEMP,\n     \"DO\" : DO,\n    \"Needle_valve_turns_open\": NEEDLE,\n    \"Comment\": COMMENT\n}\nreturn msg; \n\n",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 590,
        "y": 260,
        "wires": [
            [
                "d0f92babef87e3cf"
            ]
        ]
    },
    {
        "id": "d0f92babef87e3cf",
        "type": "gate",
        "z": "3d7d17a620f1054b",
        "name": "",
        "controlTopic": "control",
        "defaultState": "closed",
        "openCmd": "START",
        "closeCmd": "STOP",
        "toggleCmd": "toggle",
        "defaultCmd": "default",
        "statusCmd": "status",
        "persist": false,
        "storeName": "memory",
        "x": 790,
        "y": 260,
        "wires": [
            [
                "d49eb0060c02fd95"
            ]
        ]
    },
    {
        "id": "00ee66b796406c51",
        "type": "ui_button",
        "z": "3d7d17a620f1054b",
        "name": "",
        "group": "a9a3a39916ea4f91",
        "order": 40,
        "width": 2,
        "height": 1,
        "passthru": false,
        "label": "START LOG",
        "tooltip": "",
        "color": "",
        "bgcolor": "",
        "className": "",
        "icon": "",
        "payload": "START",
        "payloadType": "str",
        "topic": "control",
        "topicType": "str",
        "x": 570,
        "y": 300,
        "wires": [
            [
                "d0f92babef87e3cf"
            ]
        ]
    },
    {
        "id": "8690270298b6ed13",
        "type": "ui_button",
        "z": "3d7d17a620f1054b",
        "name": "",
        "group": "a9a3a39916ea4f91",
        "order": 45,
        "width": 2,
        "height": 1,
        "passthru": false,
        "label": "STOP LOG",
        "tooltip": "",
        "color": "",
        "bgcolor": "",
        "className": "",
        "icon": "",
        "payload": "STOP",
        "payloadType": "str",
        "topic": "control",
        "topicType": "str",
        "x": 570,
        "y": 340,
        "wires": [
            [
                "d0f92babef87e3cf"
            ]
        ]
    },
    {
        "id": "fb1bfaca77e0954a",
        "type": "inject",
        "z": "3d7d17a620f1054b",
        "name": "Log rate 5sec interval",
        "props": [
            {
                "p": "payload"
            }
        ],
        "repeat": "5",
        "crontab": "",
        "once": true,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "true",
        "payloadType": "bool",
        "x": 360,
        "y": 260,
        "wires": [
            [
                "27f2ba6bc9cd4809"
            ]
        ]
    },
    {
        "id": "d49eb0060c02fd95",
        "type": "debug",
        "z": "3d7d17a620f1054b",
        "name": "debug 319",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "statusVal": "",
        "statusType": "auto",
        "x": 950,
        "y": 260,
        "wires": []
    },
    {
        "id": "a9a3a39916ea4f91",
        "type": "ui_group",
        "name": "",
        "tab": "6ebca4f8bdecdb95",
        "order": 1,
        "disp": true,
        "width": 28,
        "collapse": false,
        "className": ""
    },
    {
        "id": "6ebca4f8bdecdb95",
        "type": "ui_tab",
        "name": "WP1",
        "icon": "dashboard",
        "order": 14,
        "disabled": false,
        "hidden": false
    }
]

[jbudd]

You are using flow context variables to hold your incoming data and make it available to the CSV logging code.
That's fair enough, though I personally try to minimise use of context variables, instead passing data along the flow wires as msg.payload properties.
However, it obscures how the data gets into the flow and it's format.
Do you request it from the sensor or does it arrive automatically?

If you request it, the chosen interval can control the rate of requests.
If it arrives automatically, the interval has to control how often you pass a set of readings through to the logging code.

[CADonkey]

The data (to be logged) is coming into Node-Red from different sources. Some is from an OPCUA client from an OPC Server in a separate windows application, there is a mass balance value over a serial link, a pump RPM speed over another serial link, and then another serial link reading in a bunch of analogue sensor values and bit-states.

Some of that data is graphed and fed to meters constantly so that the user can see the effect of parameter changes without lag.

The test runs might vary between a few hours, up to 4-5 days, so the user wants to be able to set the log-rate between 5 seconds up to 10minutes.

If there isn't s standard trigger node, or loop node that allows a 'codeable' time-base entry, then I'm guessing that I probably want a function node that perpetually loops with a delay linked to a var, and sends the payload on completion (before it loops again)

Many thanks for the help so far.

[gerry]

There are quite a few variable or programmable timers, you can search for them but you'll also have the issue of how to program them. If it's user programmable you'll have to put up a page to set the time. One I use is something something mytimer ( can't remember exactly) but there are others. Not at home but this seems to be a common scenario for using node red. If you get stuck I can probably send a code snippet, let me know.

[Colin]

If you are looking for, effectively, an Inject node with variable rate then you can use a Trigger node for that.

[{"id":"2712d95f54166b99","type":"trigger","z":"4d49653279fea87a","name":"Variable inject","op1":"1","op2":"0","op1type":"str","op2type":"str","duration":"-5","extend":false,"overrideDelay":true,"units":"s","reset":"","bytopic":"all","topic":"topic","outputs":1,"x":320,"y":620,"wires":[["83cc98a3c3f3898d"]]},{"id":"4ea76eac20a944f1","type":"inject","z":"4d49653279fea87a","name":"2 seconds","props":[{"p":"delay","v":"2000","vt":"num"},{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"str","x":100,"y":580,"wires":[["2712d95f54166b99"]]},{"id":"1bdb20685c4cc2a7","type":"inject","z":"4d49653279fea87a","name":"5 seconds","props":[{"p":"delay","v":"5000","vt":"num"},{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"str","x":100,"y":620,"wires":[["2712d95f54166b99"]]},{"id":"0ff24da16d442882","type":"inject","z":"4d49653279fea87a","name":"10 seconds","props":[{"p":"delay","v":"10000","vt":"num"},{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"str","x":110,"y":660,"wires":[["2712d95f54166b99"]]},{"id":"2f5b65efab85979d","type":"inject","z":"4d49653279fea87a","name":"Stop","props":[{"p":"reset","v":"true","vt":"bool"},{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"str","x":150,"y":720,"wires":[["2712d95f54166b99"]]},{"id":"83cc98a3c3f3898d","type":"debug","z":"4d49653279fea87a","name":"debug 105","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":510,"y":620,"wires":[]}]

[jbudd]

@CADonkey

The data (to be logged) is coming into Node-Red from different sources.

OK so your flow context variables are a sensible way to accumulate the data.
The alternative is to join multiple messages into a single one with a Join node.

You can get the frequency value from the dashboard as seconds, from 5 to 600.
SInce msg.delay has to be milliseconds, multiply the value by 1000.
Use this as the input to @Colin's Variable inject.

[{"id":"47e2e8f240dae255","type":"ui_slider","z":"3d7d17a620f1054b","name":"Frequency","label":"Frequency (sec)","tooltip":"","group":"a9a3a39916ea4f91","order":2,"width":"6","height":"2","passthru":false,"outs":"end","topic":"topic","topicType":"msg","min":"5","max":"600","step":"10","className":"","x":90,"y":220,"wires":[["c94ce83995b7b9ed"]]},{"id":"c94ce83995b7b9ed","type":"change","z":"3d7d17a620f1054b","name":"","rules":[{"t":"set","p":"delay","pt":"msg","to":"payload * 1000","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":260,"y":220,"wires":[[]]},{"id":"a9a3a39916ea4f91","type":"ui_group","name":"","tab":"6ebca4f8bdecdb95","order":1,"disp":true,"width":28,"collapse":false,"className":""},{"id":"6ebca4f8bdecdb95","type":"ui_tab","name":"WP1","icon":"dashboard","order":14,"disabled":false,"hidden":false}]

[CADonkey]

Hi Colin,

Ah yes, I had looked at a trigger node but hadn't spotted the override trigger with msg.delay checkbox. That looks to be the easiest way to do what I want.

Many thanks.

[CADonkey]

Hi Jerry,

Thanks for that, I was just trying the timeout node, which can be triggered with a 'tickle' inpuit, and looked like I could do it with that, but Colin has just pointed out that the trigger node value can be overridden with msg.delay, so that might be a bit simpler way of doing it.

I'll try it tomorrow, and should hopefully get it working how I want now with everyone's input.