Problem opening file written by write node

General
1

I'm putting this here as I'm not sure where else it would go!
I have three flows that each write to their own .txt file.
two of the flows I can open in the text editor on my raspberry pi, but when opening the third it tells me that the 'document was not UTF-8 valid' and has an 'invalid byte sequence in conversion input'
However, when I transfer the file to my windows computer it opens perfectly (either when opened in Notepad++, or read by a python script)

I have checked the json code for each write node and it is the same:

[{"id":"a6ee07c5.d985b","type":"file","z":"cd4a564c.a0337","name":"","filename":"/home/pi/V3_data/data.txt","appendNewline":true,"createDir":false,"overwriteFile":"false","encoding":"none","x":730,"y":240,"wires":[[]]}]

[{"id":"e07a7a67.70a8c8","type":"file","z":"d7a12325.a3cce","name":"","filename":"/home/pi/V3_data/davis_data.txt","appendNewline":true,"createDir":false,"overwriteFile":"false","encoding":"none","x":895,"y":360,"wires":[[]],"l":false}]

[{"id":"ef190b7e.fddf98","type":"file","z":"83d19a9a.f2f1e8","name":"","filename":"/home/pi/V3_data/ws2_5min.txt","appendNewline":true,"createDir":false,"overwriteFile":"false","encoding":"none","x":730,"y":260,"wires":[[]]}]

all have 'encoding:none'
the file that won't open is the one written to 'data.txt'
can anyone shed light on this?

2

It might have something to do with the data you are writting to the file. Is it different from the other files?

I suggest to place a debug node before all the write to file nodes. And check for differences in the payload.

Can you share (part) of the flow?

1 Like
3

This is the confusing part, I used to be able to open the file, but since running an update on my pi, it seems I can't!
here are the flows: this is the one that wont open -

[{"id":"cd4a564c.a0337","type":"tab","label":"pennal data record","disabled":false,"info":""},{"id":"891aa09b.ae1158","type":"mqtt in","z":"cd4a564c.a0337","name":"pennal","topic":"#","qos":"2","datatype":"json","broker":"eec8d6a7.9e07f8","nl":false,"rap":true,"rh":0,"x":70,"y":240,"wires":[["5f19cb45.2b2904","31d99252.2ad976"]]},{"id":"5f19cb45.2b2904","type":"function","z":"cd4a564c.a0337","name":"","func":"\n//var rssi_b\n//var rssi_o\n//\n//\n//if (msg.payload.uplink_message.rx_metadata[0].gateway_ids == \"pennal-blue-gw\") {\n//rssi_b= msg.payload.uplink_message.rx_metadata[0].rssi;\n//rssi_o = msg.payload.uplink_message.rx_metadata[1].rssi;\n//} else {\n//rssi_b= msg.payload.uplink_message.rx_metadata[1].rssi;\n//rssi_o = msg.payload.uplink_message.rx_metadata[0].rssi;\n//}\n//payload= {\"time\":payload.received_at,\"devid\":msg.payload.end_device_ids.device_id,\"rainmm\":msg.payload.uplink_message.decoded_payload.rainmm,\"battery\":payload.uplink_message.decoded_payload.voltage,\"Rssi_b\":rssi_b,\"Rssi_o\":rssi_o};\n\npayload={\"time\":msg.payload.received_at,\"devid\":msg.payload.end_device_ids.device_id,\"rainmm\":msg.payload.uplink_message.decoded_payload.rainmm,\"battery\":msg.payload.uplink_message.decoded_payload.voltage};\nmsg.payload=JSON.stringify(payload);\nmsg.payload=payload;\n\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":300,"y":240,"wires":[["68ec8ce1.e1e8cc","9ff563d2.3fb45"]]},{"id":"68ec8ce1.e1e8cc","type":"csv","z":"cd4a564c.a0337","name":"csv ouit","sep":",","hdrin":"","hdrout":"none","multi":"one","ret":"\\n","temp":"time, devid, rainmm, battery","skip":"0","strings":true,"include_empty_strings":"","include_null_values":"","x":500,"y":240,"wires":[["a6ee07c5.d985b"]]},{"id":"a6ee07c5.d985b","type":"file","z":"cd4a564c.a0337","name":"","filename":"/home/pi/V3_data/data.txt","appendNewline":true,"createDir":false,"overwriteFile":"false","encoding":"none","x":730,"y":240,"wires":[[]]},{"id":"9ff563d2.3fb45","type":"debug","z":"cd4a564c.a0337","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":490,"y":360,"wires":[]},{"id":"31d99252.2ad976","type":"debug","z":"cd4a564c.a0337","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":220,"y":340,"wires":[]},{"id":"eec8d6a7.9e07f8","type":"mqtt-broker","name":"matts api","broker":"eu1.cloud.thethings.network","port":"8883","tls":"b3f07811.bb1148","clientid":"","usetls":true,"protocolVersion":"4","keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","birthMsg":{},"closeTopic":"","closeQos":"0","closePayload":"","closeMsg":{},"willTopic":"","willQos":"0","willPayload":"","willMsg":{},"sessionExpiry":""},{"id":"b3f07811.bb1148","type":"tls-config","name":"","cert":"","key":"","ca":"","certname":"","keyname":"","caname":"","servername":"","verifyservercert":true}]

whereas this one will open the written file:

[{"id":"83d19a9a.f2f1e8","type":"tab","label":"weather station 2","disabled":false,"info":""},{"id":"d398b224.5d6c","type":"mqtt in","z":"83d19a9a.f2f1e8","name":"","topic":"data","qos":"1","datatype":"auto","broker":"f0a694ef.dc29c8","nl":false,"rap":true,"rh":0,"x":210,"y":260,"wires":[["a6e89aa5.d2d45","6c71def3.a9dcc"]]},{"id":"a6e89aa5.d2d45","type":"gate","z":"83d19a9a.f2f1e8","name":"gate demo","controlTopic":"control","defaultState":"closed","openCmd":"open","closeCmd":"close","toggleCmd":"toggle","defaultCmd":"default","statusCmd":"status","persist":false,"x":470,"y":260,"wires":[["d8fb6d9b.e645a8","ef190b7e.fddf98"]]},{"id":"8fe5ea82.c9dd9","type":"inject","z":"83d19a9a.f2f1e8","name":"open","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"300","crontab":"","once":true,"onceDelay":0.1,"topic":"control","payload":"open","payloadType":"str","x":230,"y":220,"wires":[["a6e89aa5.d2d45"]]},{"id":"d82c3dd.81f8d4","type":"inject","z":"83d19a9a.f2f1e8","name":"close","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"300","crontab":"","once":true,"onceDelay":"15","topic":"control","payload":"close","payloadType":"str","x":230,"y":300,"wires":[["a6e89aa5.d2d45"]]},{"id":"d8fb6d9b.e645a8","type":"debug","z":"83d19a9a.f2f1e8","name":"output","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":630,"y":360,"wires":[]},{"id":"ef190b7e.fddf98","type":"file","z":"83d19a9a.f2f1e8","name":"","filename":"/home/pi/V3_data/ws2_5min.txt","appendNewline":true,"createDir":false,"overwriteFile":"false","encoding":"none","x":730,"y":260,"wires":[[]]},{"id":"6c71def3.a9dcc","type":"debug","z":"83d19a9a.f2f1e8","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":420,"y":140,"wires":[]},{"id":"f0a694ef.dc29c8","type":"mqtt-broker","name":"","broker":"f0874374ae47402d8841ebf07b2d36d6.s1.eu.hivemq.cloud","port":"8883","tls":"b3f07811.bb1148","clientid":"","usetls":true,"protocolVersion":"4","keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","birthMsg":{},"closeTopic":"","closeQos":"0","closePayload":"","closeMsg":{},"willTopic":"","willQos":"0","willPayload":"","willMsg":{},"sessionExpiry":""},{"id":"b3f07811.bb1148","type":"tls-config","name":"","cert":"","key":"","ca":"","certname":"","keyname":"","caname":"","servername":"","verifyservercert":true}]

i find it odd that it won't open on the pi, but the fille can be read on the windows machine!

4

Try changing that function to ...

const payload = {
    "time": msg.payload.received_at,
    "devid": msg.payload.end_device_ids.device_id,
    "rainmm": msg.payload.uplink_message.decoded_payload.rainmm,
    "battery": msg.payload.uplink_message.decoded_payload.voltage
};
const payloadString = JSON.stringify(payload);
const utf8String = Buffer.from(payloadString, 'utf-8').toString();
msg.payload = JSON.parse(utf8String);
return msg;

this ensures whatever is coming in is gonna end up as UTF-8 when written to file

5

Thanks @Steve-Mcl , but I don't see how that will solve the issue. the flow has been running since May of this year and, as I said earlier, i used to be able to open the file on my pi. changing the function now will not affect the data already written, so will not allow me to open the file. Also as the file opens successfully on my windows computer that i use to analyse the data I think I'll just live with it. :slight_smile:

6

No it won't. The code I gave you should prevent future issues.

You can't fix the file unless you do something similar (load file into node-red as buffer, convert to utf-8 string, save back to file system)

Ps, I'm guessing you want to keep this data - kinda like a database? Perhaps you should use a database instead? Not difficult to do.

7

Yeah, lol, it's complicated! I pull the .txt file daily from my pi to my windows laptop, where it's parsed and put into a postgres table. At the time I began the project i had no clue about node red at all, let alone how to write from node red to a db.
Also, i installed notepadqq on the pi, and the file opens successfully, so problem solved!!

Thanks for your help anyway :slight_smile:

8

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.