# Read out log file and add new entries to an existing file

**URL:** <https://discourse.nodered.org/t/read-out-log-file-and-add-new-entries-to-an-existing-file/76791>\
**Category:** General\
**Tags:** docker\
**Created:** [21 March 2023 20:06 UTC](https://discourse.nodered.org/t/read-out-log-file-and-add-new-entries-to-an-existing-file/76791 "2023-03-21T20:06:26Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![halfdan](https://avatars.discourse-cdn.com/v4/letter/h/d9b06d/32.png) [@halfdan](https://discourse.nodered.org/u/halfdan)\
**Post date:** [21 March 2023 20:06 UTC](https://discourse.nodered.org/t/read-out-log-file-and-add-new-entries-to-an-existing-file/76791/1 "2023-03-21T20:06:26Z")

</div>

Hello Node Red community,

I would like to achieve the following and hope you can tell me if this is possible with node red.

I want to read a log file from a web server. There is little change here and every few days an entry is added.

A maximum of 100 entries are stored there. So that I don't lose them, I want to read out the log file once a day and add only new entries to a text file on my home server (csv format).  
Can this work?  
Thanks for your input!

My node red runs in docker and is version nodered/node-red:2.2.2-12

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [21 March 2023 21:04 UTC](https://discourse.nodered.org/t/read-out-log-file-and-add-new-entries-to-an-existing-file/76791/2 "2023-03-21T21:04:46Z")

</div>

Yes, you can do that as long as Node-RED has access to the log file. Personally, I wouldn't do it that way myself but I understand that it may be easier if you don't know any shell scripting.

To be honest though, if you don't want to lose log files, you should probably use a backup tool to keep as many versions as you need.

Just note though that docker creates a virtual environment around whatever it is running. This means that node-red inside docker is effectively running on a separate server. Working out the details of the required networking can sometimes be challenging.

---

<div class="post-metadata">

**Author:** ![halfdan](https://avatars.discourse-cdn.com/v4/letter/h/d9b06d/32.png) [@halfdan](https://discourse.nodered.org/u/halfdan)\
**Post date:** [22 March 2023 06:47 UTC](https://discourse.nodered.org/t/read-out-log-file-and-add-new-entries-to-an-existing-file/76791/3 "2023-03-22T06:47:08Z")

</div>

Hello @TotallyInformation,  
thank you for your feedback!  
Can you give me a hint what would be the appropriate nodes to compare "text a" (log file) with "text b" (backup) and append the new entries from "text a" to "text b"?  
Thanks a lot for your effort!

---

<div class="post-metadata">

**Author:** ![jbudd](https://avatars.discourse-cdn.com/v4/letter/j/5f8ce5/32.png) [@jbudd](https://discourse.nodered.org/u/jbudd)\
**Post date:** [22 March 2023 07:25 UTC](https://discourse.nodered.org/t/read-out-log-file-and-add-new-entries-to-an-existing-file/76791/4 "2023-03-22T07:25:56Z")

</div>

> [@halfdan](#):
>
> A maximum of 100 entries are stored there.

Do you mean a line is dropped from the beginning of the logfile when line 101 is added?  
Or, like most logfiles, it just keeps on growing (but slowly)?

If you are on Linux the easiest way to compare two files is with an exec node calling the command line tool **cmp newfile oldfile** which will return {code:0} if they are the same and {code: 1} if they differ.

If the only possible change to the log file is that new entries are added, the easiest way to append the entries from the new file to the old file is another command line utility **mv newfile oldfile**

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [22 March 2023 09:34 UTC](https://discourse.nodered.org/t/read-out-log-file-and-add-new-entries-to-an-existing-file/76791/5 "2023-03-22T09:34:07Z")

</div>

> [@halfdan](#):
>
> Can you give me a hint what would be the appropriate nodes to compare "text a" (log file) with "text b" (backup) and append the new entries from "text a" to "text b"?

Depends on the log file. You will need to share a few lines of the log so that we can give better answers.

---

<div class="post-metadata">

**Author:** ![halfdan](https://avatars.discourse-cdn.com/v4/letter/h/d9b06d/32.png) [@halfdan](https://discourse.nodered.org/u/halfdan)\
**Post date:** [22 March 2023 09:50 UTC](https://discourse.nodered.org/t/read-out-log-file-and-add-new-entries-to-an-existing-file/76791/6 "2023-03-22T09:50:45Z")

</div>

Hi @jbudd,  
exactly, Line Number 1 is dropped, if Line 101 gets added.

Here is an example of the csv.

Date;Duration;Energy;Price;Costs;StartReading;UID;Username  
2023-3-10 6:42:14;0:6:1;0,02;29,9;0,00598;1167,326;vehicle;vehicle

But the new file should have the entries starting from 1 to 101 and further on (no dropping of lines).

---

<div class="post-metadata">

**Author:** ![E1cid](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/e1cid/32/77971_2.png) [@E1cid](https://discourse.nodered.org/u/E1cid)\
**Post date:** [22 March 2023 10:27 UTC](https://discourse.nodered.org/t/read-out-log-file-and-add-new-entries-to-an-existing-file/76791/7 "2023-03-22T10:27:41Z")

</div>

Here is an example of how to do similar in node-red

```auto
[{"id":"5cd955ecde8bbb91","type":"inject","z":"65617ffeb779f51c","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":160,"y":4320,"wires":[["967fbe6db440f583"]]},{"id":"967fbe6db440f583","type":"template","z":"65617ffeb779f51c","name":"simulate read file_a","field":"payload","fieldType":"msg","format":"handlebars","syntax":"mustache","template":"Date;Duration;Energy;Price;Costs;StartReading;UID;Username\n2023-3-10 6:42:14;0:6:1;0,02;29,9;0,00598;1167,326;vehicle;vehicle\n","output":"str","x":330,"y":4320,"wires":[["403ec7b80e39a85c"]]},{"id":"403ec7b80e39a85c","type":"change","z":"65617ffeb779f51c","name":"","rules":[{"t":"set","p":"original_file_a","pt":"msg","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":540,"y":4320,"wires":[["6a38f80097a06eba"]]},{"id":"6a38f80097a06eba","type":"csv","z":"65617ffeb779f51c","name":"","sep":";","hdrin":true,"hdrout":"none","multi":"mult","ret":"\\n","temp":"","skip":"0","strings":true,"include_empty_strings":"","include_null_values":"","x":710,"y":4320,"wires":[["9c360c9412158332"]]},{"id":"9c360c9412158332","type":"change","z":"65617ffeb779f51c","name":"","rules":[{"t":"move","p":"payload","pt":"msg","to":"file_a","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":870,"y":4320,"wires":[["3c5db59cc227b006"]]},{"id":"3c5db59cc227b006","type":"template","z":"65617ffeb779f51c","name":"simulate read file b","field":"payload","fieldType":"msg","format":"handlebars","syntax":"mustache","template":"Date;Duration;Energy;Price;Costs;StartReading;UID;Username\n2023-3-10 6:42:14;0:6:1;0,02;29,9;0,00598;1167,326;vehicle;vehicle\n2023-3-11 6:42:15;0:6:1;0,02;29,9;0,00598;1167,326;vehicle;vehicle\n","output":"str","x":230,"y":4380,"wires":[["e411da20cbae16be"]]},{"id":"e411da20cbae16be","type":"csv","z":"65617ffeb779f51c","name":"","sep":";","hdrin":true,"hdrout":"none","multi":"mult","ret":"\\n","temp":"","skip":"0","strings":true,"include_empty_strings":"","include_null_values":"","x":390,"y":4380,"wires":[["c6d133fbef5674ac"]]},{"id":"c6d133fbef5674ac","type":"change","z":"65617ffeb779f51c","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"(\t $dates_in_file_a := $$.file_a.Date;\t$append($$.file_a,$$.payload[$not($.Date in $dates_in_file_a)][])\t)","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":560,"y":4380,"wires":[["117ae6632c1bf80e"]]},{"id":"117ae6632c1bf80e","type":"csv","z":"65617ffeb779f51c","name":"","sep":";","hdrin":true,"hdrout":"all","multi":"mult","ret":"\\n","temp":"","skip":"0","strings":true,"include_empty_strings":"","include_null_values":"","x":710,"y":4380,"wires":[["f3d15ea8be9da0d8"]]},{"id":"f3d15ea8be9da0d8","type":"debug","z":"65617ffeb779f51c","name":"file_a contains og file object\\n payload contains new updated file_a\\n original_file_a contains og csv","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":290,"y":4440,"wires":[]}]

```

I have simulated the reading of the files, these should be read file nodes. The output can be saved using a file write node.

Your example does not explain the differences between the files, best to show a example of both files (obviously not 100 entries maybe five in each). Then explain what the different lines that would be moved are and how they are identified.

---

<div class="post-metadata">

**Author:** ![halfdan](https://avatars.discourse-cdn.com/v4/letter/h/d9b06d/32.png) [@halfdan](https://discourse.nodered.org/u/halfdan)\
**Post date:** [22 March 2023 10:56 UTC](https://discourse.nodered.org/t/read-out-log-file-and-add-new-entries-to-an-existing-file/76791/8 "2023-03-22T10:56:00Z")

</div>

Hi @E1cid,

problem is, the api call of the device is currently not working, so i can't provide an example. From the manufacturer I have the info, that with this api call you should get a json file with similar content as the above posted lines.

At the moment I use the WebUI of the device every week or so and there I click "download log file as csv".

This csv delivers the posted 1st line and following 100 lines of data. When new data is generated, the oldest entry of the logfile vanishes.  
If possible, I would like to have this process automated.

I think I will experiment with your example, to get a feeling, thanks a lot!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)\
**Post date:** [21 May 2023 10:56 UTC](https://discourse.nodered.org/t/read-out-log-file-and-add-new-entries-to-an-existing-file/76791/9 "2023-05-21T10:56:04Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.