Hello,
I have a problem with the file node.
When I extract data with the file write node, everything is fine; no syntax problem. On the other hand, when I want to recover the data with the file reads node, I have syntax errors in the payload. the 'é' turns into '?'. If anyone could enlighten me.
Thank you
Can you post a small flow that shows the problem so we can try it?
What system are you running on? As on a mac I cannot reproduce
[{"id":"7131ffea.7413a8","type":"inject","z":"efbf8bf.35920f8","name":"","topic":"","payload":"joãoéßæ","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":160,"y":140,"wires":[["8c4e484d.c8b418"]]},{"id":"8c4e484d.c8b418","type":"file","z":"efbf8bf.35920f8","name":"","filename”:”/<full path to file>/test.txt","appendNewline":false,"createDir":true,"overwriteFile":"true","x":450,"y":140,"wires":[]},{"id":"f6715482.41c258","type":"debug","z":"efbf8bf.35920f8","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","x":550,"y":240,"wires":[]},{"id":"9eaa97fa.eb3b18","type":"file in","z":"efbf8bf.35920f8","name":"","filename":"/<full path to file>/test.txt","format":"utf8","chunk":false,"sendError":false,"x":350,"y":240,"wires":[["f6715482.41c258"]]},{"id":"697fa3d0.4bc9bc","type":"inject","z":"efbf8bf.35920f8","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":100,"y":240,"wires":[["9eaa97fa.eb3b18"]]}]
Produces the following in the debug panel:
_msgid: "a463efe0.17f5b"
topic: ""
payload: "joãoéßæ"
filename:
If you write a file with Node-RED file in node and then read this file with Node-RED file it should work flawlessly. The character "é" will be saved as the UTF-8 sequence of two bytes "c3a9" as expected from the UTF-8 character set.
Now, if you saved the file with another text editor it may happen that the character will be saved with another encoding, for instance, ISO-8859-1 (ISO Latin 1). In such case the character "é" will be saved as byte "e9" , which is not a valid byte in UTF-8, resulting that in Node-RED node it will be displayed as the replacement character �
1 Like
Node red runs on a raspberry with linux. I'm using the file node to save the data:
(ciel dégagé 7.7 degré 1.5 pourcent )
then I use the file in node to retrieve and read the data:
( 25/11/2018 à 17:49:22node: 57c59eab.874de8
msg.payload : string[36]
"ciel d�gag� 7.7 degr� 1.5 pourcent " )
file name: / home / pi / logs / folder name .
This testing may help to clarify ( peut-être).
What do you get on the debug panel if you change your flow with this two changes ?
1- Firstly change the config node of your file node (read) as below:
2- add this function node after the file read node
[{"id":"ee5b88b8.33e098","type":"debug","z":"1a1dea55.020d86","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":670,"y":180,"wires":[]},{"id":"8797bab8.7d5eb8","type":"function","z":"1a1dea55.020d86","name":"latin1","func":"pay = msg.payload.toString(\"latin1\");\nmsg.payload = pay;\nreturn msg;","outputs":1,"noerr":0,"x":470,"y":180,"wires":[["ee5b88b8.33e098"]]}]
That's what I get:
[110,117,97,103,101,117,120,32,32,54,46,55,32,100,101,103,114,233,32,48,46,53,32,112,111,117,114,99,101,110,116,32]
thank you for your interest in my problem
Good. It means that we do have a workaround for the issue. Missing to know why the file was saved in a way that is not usual (I can imagine a couple of reasons, but have no way to test).
This is what you will see in the debug panel:
But you need to add the function node after the file node, as below:
Explanation:
This is the code in the function node:
pay = msg.payload.toString("latin1");
msg.payload = pay;
return msg;
It will decode the data read by the file node using the 8859-1 (Latin-1) character set.
This is my testing flow. You need to change the file name and path in the file node to test again on your side.
[{"id":"87eb973e.4accd8","type":"tab","label":"Flow 1","disabled":false,"info":""},{"id":"81036dd5.c91ce","type":"file in","z":"87eb973e.4accd8","name":"Read from file","filename":"C:/Users/OCM/.node-red/static/nrfiles/coffee.txt","format":"","chunk":false,"sendError":false,"x":340,"y":140,"wires":[["3c2937f5.6c7648"]]},{"id":"f6c975f9.5bf1c8","type":"inject","z":"87eb973e.4accd8","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":160,"y":140,"wires":[["81036dd5.c91ce"]]},{"id":"39cd1e82.01f6f2","type":"debug","z":"87eb973e.4accd8","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":650,"y":140,"wires":[]},{"id":"3c2937f5.6c7648","type":"function","z":"87eb973e.4accd8","name":"latin1","func":"\npay = msg.payload.toString(\"latin1\");\nmsg.payload = pay;\nreturn msg;","outputs":1,"noerr":0,"x":510,"y":140,"wires":[["39cd1e82.01f6f2"]]}]
I tested, but it did not work:
the file is overwritten each time there is a new value of the openweathermap node
Puzzling indeed. In such case we need to debug a little further.
Could you please confirm if the function indicated with a red arrow is the one I suggested ?
Can you possibly add a debug node where it is indicated with number 3 , run the flow and show a new output from the debug panel ? I wanted to compare the outputs of debug nodes 1, 2 and 3.
Good evening,
it is the node of function that you proposed to me;
and here is the result of the debug;
debug 2 and 3 are the same.
Interesting. My best gues now is that the data is being saved wrongly so the conversion function does not work, but not having a raspberry I can not test by myself.
I wanted to know what this the default language and encoding used on your raspberry.
Google points these possible commands in the terminal. Could you please test and see what you can find ? (merci)
locale
or maybe using the command echo $LANG in the terminal ?
or perhaps you can check in this system menu:
Looks like your raspberry is configured with the default UTF-8, so nothing suspicious here.
I was not able to identify the root cause neither the solution for the reported problem.
Summary in case someone else has some clue:
1- Text from Openweathermap is nuageux 6.7 degré 0.5 pourcent
2- This text is saved with node-file
3- Binary data is saved and read as below. Note that the character é is the 17th byte which is decimal 233 , hexa e9. The assumption is that it was saved as ISO-8859-1 (ISO Latin 1) given that 0xe9 is invalid in UTF-8
[110,117,97,103,101,117,120,32,32,54,46,55,32,100,101,103,114,233,32,48,46,53,32,112,111,117,114,99,101,110,116,32]
4- The issue is that after reading the file with the node file in it appears in the debug panel as nuageux 6.7 degr� 0.5 pourcent
5- We tried to use a function node to convert from latin-1 character set to UTF-8. It works in my system (windows10) but it did not work for @fabien (raspberry pi) .
Testing flow:
[{"id":"ee5b88b8.33e098","type":"debug","z":"1a1dea55.020d86","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","x":550,"y":180,"wires":[]},{"id":"8797bab8.7d5eb8","type":"function","z":"1a1dea55.020d86","name":"latin1 - buffer","func":"msg.payload = Buffer.from([110,117,97,103,101,117,120,32,32,54,46,55,32,100,101,103,114,233,32,48,46,53,32,112,111,117,114,99,101,110,116,32]);\n\nmsg.latin1 = msg.payload.toString(\"latin1\");\nmsg.utf8 = msg.payload.toString(\"utf-8\");\n\n\nreturn msg;","outputs":1,"noerr":0,"x":350,"y":180,"wires":[["ee5b88b8.33e098","60224a4a.26e6c4","8f4ceaa9.6bd968"]]},{"id":"98ddfefc.df073","type":"inject","z":"1a1dea55.020d86","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":100,"y":180,"wires":[["8797bab8.7d5eb8"]]},{"id":"60224a4a.26e6c4","type":"ui_text","z":"1a1dea55.020d86","group":"90301f70.997cd","order":0,"width":0,"height":0,"name":"","label":"Latin1:","format":"{{msg.latin1}}","layout":"row-center","x":550,"y":240,"wires":[]},{"id":"8f4ceaa9.6bd968","type":"ui_text","z":"1a1dea55.020d86","group":"90301f70.997cd","order":0,"width":0,"height":0,"name":"","label":"UTF-8:","format":"{{msg.utf8}}","layout":"row-center","x":550,"y":280,"wires":[]},{"id":"90301f70.997cd","type":"ui_group","z":"","name":"Teste","tab":"1e2aea29.e92e76","disp":false,"width":"10","collapse":false},{"id":"1e2aea29.e92e76","type":"ui_tab","z":"","name":"LAB","icon":"dashboard"}]
Which version of node.js is @fabien running ?
Versions prior to 6,14.0 did not have the latin1 support your function node uses.
I have pushed a beta version of dashboard to master (not npm) that has a fix to try to handle binary buffers in a better manner - ie not force them to utf8 when displaying text,
So it should now just handle the binary buffer msg.payload direct.
You can get it by
cd ~/.node-red && npm i node-red/node-red-dashboard
Let me know if it helps / makes things worse - etc
1 Like
Did you restart node-red and force a full refresh of your browser?
1 Like
