Node-red.log in Temp Folder

General
1

I would like to address the issue regarding the Node-red.log file size in the Temp folder located on the C-drive. The file size has been consistently increasing, reaching sizes as high as 60 GB and continues to grow. We need to implement measures to manage this growth and ensure its efficient management.

To tackle this, I propose implementing a strategy to split the Node-red.log file on a daily basis. By doing so, we can efficiently manage the file size and delete unnecessary logs as needed. This approach will help us maintain system efficiency while ensuring essential log data is retained for analysis and troubleshooting purposes.??????

2

You did not state so, but I can infer from the mention of "C-drive" that you are running node-red on windows.

Node-RED typically does NOT log to file on windows.

At a guess you have either redirected the output of Node-RED to a log file or you are using pm2 or some other service runner?

In my old day job, I would run node-red via nssm, pipe the node-red output to file and have nssm take care of file rotation.

3

Would you like to post a section of the log? It sounds as if you have something spamming the log with messages. 60GB of log is vast. My system logs a few messages a minute. I estimate that it would take about 240 years to get to 60GB.

Copy/paste please, not screenshot.

Most running node-red use Linux (I think), where log rotation is, of course, handled automatically.

4

Sorry for delayed input here.

@dceejay's instructions Running Node-RED as a service on Windows using NSSM do indeed tell Node-red to log to c:\temp/node-red.log:

nssm set Node-RED AppParameters "-u c:\Users\\"%USERNAME%"\.node-red > c:\temp\node-red.log"

Can you explain what you mean by "pipe the NR output" in the context of Windows - is Dave's > ok here? (I know the difference on Linux!)
How do we set up file rotation on WIndows?

5

something.exe > log.file - this is piping the output of something.exe

Well, it will work, but I think it is only for STDOUT (it might include both, i forget) but what i am sure of is it locks the file (i.e. cant be deleted or rotated as the file handle is held open as long as the process lives)

The better way, when using NSSM is to set up both STDOUT and STDERR in the i/o tab (see docs: https://nssm.cc/usage)

I dont think Windows has this by default (happy to be told otherwise) but NSSM has a tab named File rotation - it goes hand in hand with the "I/O" tab

6

Command line pipes are the same on Windows (as it is a POSIX compatible filing system) as on Linux. I'd have to look up the syntax to get the stderr output as well, something like adding &2>1?

But yes, telling nssm to do it properly would be better - assuming you can get nssm to work.

It does not. The simplest approach may be to write a script that copies the contents of the log file and then deletes the content. But that assumes that nssm/node.js didn't lock the file preventing another task from accessing or writing to it.

If this seems odd, it is because Windows anticipates that you will use the Windows event logging service for all of this. The Windows Event Logs are managed internally by Windows.

By the way, I asked ChatGPT4o "How can I make a node.js app running on Windows 11 log to the windows event logs?" This link will take you to the answer which involves using the Winston logging module. I think there might be a node for this but if not, you can set up a custom log output that would use Winston.

https://chatgpt.com/share/0d86315b-3cb7-412e-b6d4-459a1de8e70c

That's if you wanted to use Windows Event Logs of course.

However, you could also use a custom logger to do the log rotation for you as well.

7

I've never had any problems with it.

So long as you have permitted the account to run as a service and ensure the account has all the files in place, it works flawlessly.

8

Plenty of stuff soaring over my head there but I'm happy to give context to this

In Linux > is redirection (to a file or file descriptor)
To send both stdout and stderr to the same file you would use command >filename 2>&1
A pipe | on the other hand sends the stdout of one command to the stdin of another command as in ls | wc -l.
I asked in order to be certain that Steve's use of "pipe the output" was the same as Dave's use of redirection. :grin:

9

Knew it was something like that :rofl:

Hey, at my age, I'm lucky to remember anything! :grinning:

But in fact, nssm does have some log rotation built in, the downside is that you have to restart the service to trigger it fully reliably. But that should be possible to do. Or you can see if you can trigger on a set size with the service running, it might work.

Alternatively, if you don' mind some additional overhead, use PM2 and just get it to start at windows startup - or turn it into a service instead of direct Node-RED. PM2 has its own logs and there is a log rotate extension for it.

10

Hmm yes it does but it's not log rotation as we know it Jim.

It seems to move the file to a timestamped backup each time it's started.
No evidence of deleting old files. Which, frankly, is a slow burning disaster.
Though to be fair I only experimented with nssm restart node-red 10 times in a couple of minutes.

The Node-red write-file node cannot delete the currently active logfile but it does get away with overwriting the file with a simple payload. It can delete the backups.

My head is spinning.

1 Like
11

Options, options, the joy and curse of computing :grin:

1 Like
12

Wouldn't it be simpler to switch to Linux in a Pi or an old laptop for the node-red server, and keep Windows for the Desktop machine, which is what it is reasonably good at?

13

Well the machine causing me strife is a PC in another country.
I can from time to time use TeamViewer to poke around but I have not been able to identify a cause for it's alleged shutdowns.
So I think the best option is to install Node-red and send what telemetry I can in the hope of spotting a clue.
I would like to be sure that I'm not storing up future full file system errors.
At this end there will indeed be a Pi and more Node-red.

14

If your specific requirement is to have node red logging information to a rotated file then have a look at node-red-contrib-flogger which will log to file and includes rotation.

@Mohamed_1711 perhaps that will help with your problem too.

15

My specific concern is that after setting up Node-red as a service according to the documentation, there is a logfile which can only grow and no obvious mechanism to constrain it.

Sorry but I don't see the relevance of a second parallel logging system. Are you saying that flogger can manage Node-red's console output?

16

What I did after setting up NSSM and using the I/O and File Rotation features was a simple 2am cron-plus trigger and fs-ops nodes to scan for *.log files modified over older than 1 week & delete them.

image
image703Ă—129 12.1 KB

[{"id":"c56719964f3478ec","type":"cronplus","z":"e532e271879e8af2","name":"2 am","outputField":"payload","timeZone":"","storeName":"","commandResponseMsgOutput":"output1","defaultLocation":"","defaultLocationType":"default","outputs":1,"options":[{"name":"2am","topic":"cleanup","payloadType":"str","payload":"c:/temp","expressionType":"cron","expression":"0 0 2 * * *","location":"","offset":"0","solarType":"all","solarEvents":"sunrise,sunset"}],"x":150,"y":140,"wires":[["31e557cd9528a4df"]]},{"id":"779a432bdf38d666","type":"fs-ops-dir","z":"e532e271879e8af2","name":"","path":"path","pathType":"msg","filter":"filter","filterType":"msg","dir":"files","dirType":"msg","x":600,"y":140,"wires":[["de0f6974f85ca50f"]]},{"id":"de0f6974f85ca50f","type":"fs-ops-stats","z":"e532e271879e8af2","name":"","path":"path","pathType":"msg","filename":"files","filenameType":"msg","stats":"stats","sizeType":"msg","x":740,"y":140,"wires":[["c8b79edc9e21b82a"]]},{"id":"31e557cd9528a4df","type":"change","z":"e532e271879e8af2","name":"Setup path and filter","rules":[{"t":"set","p":"path","pt":"msg","to":"c:/temp","tot":"str"},{"t":"set","p":"filter","pt":"msg","to":"*.txt","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":420,"y":140,"wires":[["779a432bdf38d666"]]},{"id":"c8b79edc9e21b82a","type":"function","z":"e532e271879e8af2","name":"get files  modified more than 7 days ago","func":"const MAX_AGE_DAYS = 7\nconst MAX_AGE_MS = MAX_AGE_DAYS * 24 * 60 * 60 * 1000\nconst FILES = msg.files\nconst now = new Date()\nmsg.oldFiles = msg.stats.filter((stat, index)=> {\n    stat.file = FILES[index]\n    const modifiedTime = new Date(stat.mtime)\n    stat.ageMs = now.valueOf() - modifiedTime.valueOf()\n    if(stat.ageMs > MAX_AGE_MS) {\n        return true\n    }\n})\nmsg.filesToDelete = msg.oldFiles.map((stat) => stat.file)\nreturn msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":480,"y":200,"wires":[["76d525e420092f1b"]]},{"id":"76d525e420092f1b","type":"fs-ops-delete","z":"e532e271879e8af2","name":"","path":"path","pathType":"msg","filename":"filesToDelete","filenameType":"msg","x":750,"y":200,"wires":[[]]}]
17

Sure there are work-arounds using contrib nodes such as node-red-contrib-fs-ops (mine uses node-red-contrib-filesystem).

The important point is that the documentation gives bad advice.

18

What doc & what bad advice?

19

I am sorry, did my first post in this thread not make that clear?

Why is that advice bad? Because it results in a file which will continually grow.
@Mohamed_1711 reports an example where it reached 60GB.

That advice is not in https://nodered.org/docs/getting-started/windows which says to use pm2 (no idea if that results in a problem). But the fact that it is in @dceejay's github gives it a certain authority.

20

No, there are links to NSSM, Setting up NSSM and Daves info - I was unsure which one you were referring to.

TBF, in the context of NSSM, it is not "great" advice for a NR that would output GBs of logs between restarts, but perfectly fine advice to get up and running with logging.

To clarify that ^ point, dave specified the use of the > redirection operator which, in DOS, will create a NEW log file upon restarting the process (>> is used to append). So since windows regularly needs restarts (for updates etc) it is not terrible advice). Had Daves info said use .node-red >> c:\temp\node then i would have agreed with you. It all depends on the situation.

The "better" (subjective) way is the I/O setup on NSSM (as mentioned earlier)