HTTP request GET - JSON parsing error

[bonito]

Hi
I am having trouble to get JSON content into Node-red. I think that I have identified the problem is in HTTP request.
If I set to as parsed JSON object. It returns as JSON parse error.
If I set it to retuns as UTF-8 and try to use Json node to parse it It says "Unexpected token  in JSON at position 0"
The Json output seems OK to me but it is not parse corectly. and return as string with 1000+ characters.
Would you be able to help?
Bewst regards

This is the output of HTTP request:

[Colin]

I think if you feed it into a JSON node instead, it will tell you where the problem is.

[bonito]

Hi Colin

I did feed it into JSON node, and the problem is "unexpected token in JSON at position 0".

I have made it successfully read and parse in Python. Which proofs that the string is OK. What I did I request HTTP get and store the result to file and than read the file in Python. And all is fine.

The problem is in parsing in Node-red.

Thanks for trying to help

[zenofmud]

That is a long string (10512). What I would do for debugging is sent that payload to a file write node to dump it into a txt file. That way you could share the actual data on the forum for people to take a look at.

[Colin]

@bonito

unexpected token in JSON at position 0

That means that the first character msg.payload is not a {. Perhaps there is a non-printing character on the front. Is debug 4 connected to the same node output as the JSON node?

I am assuming that you are not using an ancient version of node-red.

[bonito]

Hi
I thougth that the first character is wrong or there is something else. I exported it but there is nothing visible.

I dowloaded the Node red only couple of weeks ago, so I do not think so that it is old version. v3.0.2

Unfortunately I can not download the full object as zenofmud suggested, because I am new user.

Yes debug 4 is directly on the output of http request as well as the Json node.

Thank you very much for trying to help.

[Colin]

Feed the message containing the string into a function containing

msg.payload = Buffer.from(msg.payload)
return msg;

and then to a debug node. You will then be able to see exactly what is at the start of the string.

[bonito]

Hi Colin,

It was interesting now there is at begining of string this

{
"Monitor": {
"DeviceInfo": {
"D

But when I use a change node to search for { and replace with { it does nothing, and the string is instead of 10511 characters long 10518 characters long.

But I will have a few more goes with this function.

Thank you for the help so far

[zenofmud]

When I put the data you sent me into a inject node as a type string and send it thru the json node and then to a debug node, it comes as an object.

In addition, if I send it straight to a debug node, the string is 7,491 characters where yours is 10,512 characters so it is not the data you are having an issue with.

[bonito]

Paul,

I do not understand I am attaching the file which was extracted rigth at the minute and says 10511.
loging.json (10.3 KB)

This time I did not change extension or open in any editor proror to attaching, still same 10511.

I do not understand.

I tried the exact same file which I sent to you, I have 10520 character and it does not work. I am lost even more.

Can you send the file back to me if it get somehow formated?

Notepad++ says document length10517 and 341 lines and 536 words, characters without line endings 9833. The file which I sent first.

The document which I have just attached Chracters (withput line endings) :9832, Words: 536, Lines:341, Document length: 10516.

I am using Windows 7 but the node red runs on Linux Raspberry Pi

[zenofmud]

That file is valid json and shows me no errors.
What version of NR and node.js are you running? (you can get it from the startup log)

Attached is a flow where the inject node contains the login.json file which when I look at it it has 10,177 characters and takes up 12K on disk - I am running on a Mac. Note that in the json node the option Format JSON string is NOT checked. This makes the output smaller and may be why you are seeing a size of 10520.

[{"id":"d8457a23b86d841a","type":"debug","z":"024fd26097e68de9","name":"debug 1","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":660,"y":80,"wires":[]},{"id":"e9eb24911f860aca","type":"json","z":"024fd26097e68de9","name":"","property":"payload","action":"","pretty":false,"x":410,"y":140,"wires":[["b2753626b6cb39b7","111513e462ba8ae2"]]},{"id":"b2753626b6cb39b7","type":"debug","z":"024fd26097e68de9","name":"debug 2","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":660,"y":140,"wires":[]},{"id":"38b0618e8c7384c3","type":"debug","z":"024fd26097e68de9","name":"debug 3","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":660,"y":200,"wires":[]},{"id":"44adfe2b8a45c043","type":"catch","z":"024fd26097e68de9","name":"","scope":null,"uncaught":false,"x":420,"y":280,"wires":[["6f8f704c32106735"]]},{"id":"111513e462ba8ae2","type":"json","z":"024fd26097e68de9","name":"","property":"payload","action":"","pretty":true,"x":410,"y":200,"wires":[["38b0618e8c7384c3"]]},{"id":"6f8f704c32106735","type":"debug","z":"024fd26097e68de9","name":"debug 4","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":660,"y":280,"wires":[]},{"id":"55e37b2e2679c0d1","type":"inject","z":"024fd26097e68de9","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"Monitor\":{\"DeviceInfo\":{\"DeviceName\":\"TCW241\",\"HostName\":\"TCW241\",\"ID\":\"5C:32:C5:00:9E:C2\",\"FwVer\":\"TCW241-v1.240\",\"MnfInfo\":\"www.teracomsystems.com\",\"SysContact\":\"Petr\",\"SysName\":\"Garaze Relay\",\"SysLocation\":\"Garaze Anglie\"},\"S\":{\"S1\":{\"description\":\"Garaz\",\"id\":\"28245C75D0013C29\",\"item1\":{\"value\":\"4.5\",\"unit\":\"°C\",\"alarm\":\"0\",\"min\":\"-40.0\",\"max\":\"85.0\",\"hys\":\"8.5\"},\"item2\":{\"value\":\"---\",\"unit\":\"---\",\"alarm\":\"0\",\"min\":\"---\",\"max\":\"---\",\"hys\":\"---\"}},\"S2\":{\"description\":\"Sklenik\",\"id\":\"28502E75D0013C83\",\"item1\":{\"value\":\"1.3\",\"unit\":\"°C\",\"alarm\":\"0\",\"min\":\"-40.0\",\"max\":\"45.0\",\"hys\":\"4.0\"},\"item2\":{\"value\":\"---\",\"unit\":\"---\",\"alarm\":\"0\",\"min\":\"---\",\"max\":\"---\",\"hys\":\"---\"}},\"S3\":{\"description\":\"Conservator\",\"id\":\"28AE7C75D0013CF4\",\"item1\":{\"value\":\"8.6\",\"unit\":\"°C\",\"alarm\":\"0\",\"min\":\"-40.0\",\"max\":\"45.0\",\"hys\":\"8.5\"},\"item2\":{\"value\":\"---\",\"unit\":\"---\",\"alarm\":\"0\",\"min\":\"---\",\"max\":\"---\",\"hys\":\"---\"}},\"S4\":{\"description\":\"West\",\"id\":\"2894DA75D0013C79\",\"item1\":{\"value\":\"2.6\",\"unit\":\"°C\",\"alarm\":\"0\",\"min\":\"-40.0\",\"max\":\"85.0\",\"hys\":\"8.5\"},\"item2\":{\"value\":\"---\",\"unit\":\"---\",\"alarm\":\"0\",\"min\":\"---\",\"max\":\"---\",\"hys\":\"---\"}},\"S5\":{\"description\":\"East\",\"id\":\"283E2F75D0013C9E\",\"item1\":{\"value\":\"3.8\",\"unit\":\"°C\",\"alarm\":\"0\",\"min\":\"-40.0\",\"max\":\"85.0\",\"hys\":\"8.5\"},\"item2\":{\"value\":\"---\",\"unit\":\"---\",\"alarm\":\"0\",\"min\":\"---\",\"max\":\"---\",\"hys\":\"---\"}},\"S6\":{\"description\":\"S6\",\"id\":\"0000000000000000\",\"item1\":{\"value\":\"---\",\"unit\":\"---\",\"alarm\":\"0\",\"min\":\"---\",\"max\":\"---\",\"hys\":\"---\"},\"item2\":{\"value\":\"---\",\"unit\":\"---\",\"alarm\":\"0\",\"min\":\"---\",\"max\":\"---\",\"hys\":\"---\"}},\"S7\":{\"description\":\"S7\",\"id\":\"0000000000000000\",\"item1\":{\"value\":\"---\",\"unit\":\"---\",\"alarm\":\"0\",\"min\":\"---\",\"max\":\"---\",\"hys\":\"---\"},\"item2\":{\"value\":\"---\",\"unit\":\"---\",\"alarm\":\"0\",\"min\":\"---\",\"max\":\"---\",\"hys\":\"---\"}},\"S8\":{\"description\":\"S8\",\"id\":\"0000000000000000\",\"item1\":{\"value\":\"---\",\"unit\":\"---\",\"alarm\":\"0\",\"min\":\"---\",\"max\":\"---\",\"hys\":\"---\"},\"item2\":{\"value\":\"---\",\"unit\":\"---\",\"alarm\":\"0\",\"min\":\"---\",\"max\":\"---\",\"hys\":\"---\"}}},\"AI\":{\"AI1\":{\"description\":\"PSU 24V\",\"value\":\"23.98\",\"unit\":\"V\",\"multiplier\":\"1.000\",\"offset\":\"0.000\",\"alarm\":\"0\",\"min\":\"20.00\",\"max\":\"30.00\",\"hys\":\"0.30\"},\"AI2\":{\"description\":\"Auxilarygaraz1\",\"value\":\"0.03\",\"unit\":\"V\",\"multiplier\":\"1.000\",\"offset\":\"0.000\",\"alarm\":\"1\",\"min\":\"0.00\",\"max\":\"60.00\",\"hys\":\"0.10\"},\"AI3\":{\"description\":\"Auxilary garaz\",\"value\":\"0.03\",\"unit\":\"V\",\"multiplier\":\"1.000\",\"offset\":\"0.000\",\"alarm\":\"1\",\"min\":\"0.00\",\"max\":\"60.00\",\"hys\":\"1.00\"},\"AI4\":{\"description\":\"Auxilarygaraze3\",\"value\":\"0.03\",\"unit\":\"V\",\"multiplier\":\"1.000\",\"offset\":\"0.000\",\"alarm\":\"1\",\"min\":\"0.00\",\"max\":\"60.00\",\"hys\":\"1.00\"}},\"VI\":{\"VI1\":{\"description\":\"Virtual Input 1\",\"value\":\"---\",\"unit\":\"---\",\"multiplier\":\"1.000\",\"offset\":\"0.0000\",\"alarm\":\"0\",\"min\":\"---\",\"max\":\"---\",\"hys\":\"---\"},\"VI2\":{\"description\":\"Virtual Input 2\",\"value\":\"---\",\"unit\":\"---\",\"multiplier\":\"1.000\",\"offset\":\"0.0000\",\"alarm\":\"0\",\"min\":\"---\",\"max\":\"---\",\"hys\":\"---\"},\"VI3\":{\"description\":\"Virtual Input 3\",\"value\":\"---\",\"unit\":\"---\",\"multiplier\":\"1.000\",\"offset\":\"0.0000\",\"alarm\":\"0\",\"min\":\"---\",\"max\":\"---\",\"hys\":\"---\"},\"VI4\":{\"description\":\"Virtual Input 4\",\"value\":\"---\",\"unit\":\"---\",\"multiplier\":\"1.000\",\"offset\":\"0.0000\",\"alarm\":\"0\",\"min\":\"---\",\"max\":\"---\",\"hys\":\"---\"}},\"DI\":{\"DI1\":{\"description\":\"Alarm \",\"value\":\"OFF\",\"valuebin\":\"0\",\"alarmState\":\"ON\",\"alarm\":\"0\"},\"DI2\":{\"description\":\"Dvere\",\"value\":\"CLOSED\",\"valuebin\":\"0\",\"alarmState\":\"OPEN\",\"alarm\":\"0\"},\"DI3\":{\"description\":\"Vrata\",\"value\":\"OPEN\",\"valuebin\":\"1\",\"alarmState\":\"OPEN\",\"alarm\":\"1\"},\"DI4\":{\"description\":\"Digital Input 4\",\"value\":\"OPEN\",\"valuebin\":\"1\",\"alarmState\":\"CLOSED\",\"alarm\":\"0\"}},\"R\":{\"R1\":{\"description\":\"Zalevani\",\"value\":\"OFF\",\"valuebin\":\"0\",\"pulseWidth\":\"180.0\",\"control\":\"Function 1\"},\"R2\":{\"description\":\"Alarm remote\",\"value\":\"OFF\",\"valuebin\":\"0\",\"pulseWidth\":\"0.2\",\"control\":\"0\"},\"R3\":{\"description\":\"Garaze PSU\",\"value\":\"OFF\",\"valuebin\":\"0\",\"pulseWidth\":\"0.3\",\"control\":\"0\"},\"R4\":{\"description\":\"Relay 4\",\"value\":\"OFF\",\"valuebin\":\"0\",\"pulseWidth\":\"0.4\",\"control\":\"0\"}},\"HTTPPush\":{\"Key\":\"\",\"PushPeriod\":\"1800\"},\"hwerr\":\"\",\"Alarmed\":\"1\",\"Scannig\":\"\",\"Time\":{\"Date\":\"08.02.2023\",\"Time\":\"22:55:24\"}}}","payloadType":"json","x":410,"y":80,"wires":[["d8457a23b86d841a","e9eb24911f860aca"]]}]

Please try it and see what you get.

[E1cid]

I see the offending chars if i load the loging.json using file node and output binary.

How is the file created, do you have any control over it..

I can remove the offending chars with a JSONata expression

"{" & substringAfter($$.payload, "{")

eg.

[{"id":"3641dacefc1e2261","type":"inject","z":"da8a6ef0b3c9a5c8","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":120,"y":4780,"wires":[["4c02c3c535cfe9fd"]]},{"id":"4c02c3c535cfe9fd","type":"file in","z":"da8a6ef0b3c9a5c8","name":"","filename":".node-red/loging.json","filenameType":"str","format":"utf8","chunk":false,"sendError":false,"encoding":"none","allProps":false,"x":340,"y":4780,"wires":[["038d382a28b70294","c75d41fdfaeeafaa"]]},{"id":"038d382a28b70294","type":"change","z":"da8a6ef0b3c9a5c8","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"\"{\" & $substringAfter($$.payload, \"{\")","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":480,"y":4840,"wires":[["c75d41fdfaeeafaa"]]},{"id":"c75d41fdfaeeafaa","type":"json","z":"da8a6ef0b3c9a5c8","name":"","property":"payload","action":"","pretty":false,"x":670,"y":4840,"wires":[["5a4aa4835d6d0dad"]]},{"id":"5a4aa4835d6d0dad","type":"debug","z":"da8a6ef0b3c9a5c8","name":"debug 231","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":630,"y":4880,"wires":[]}]

output

[zenofmud]

@E1cid Interesting! By inserting the data from that file into an inject node I didn't hit the error but storing it into a file and reading it in, I do see the error.

@bonito If you don't want to use JSONata, you could install node-red-contrib-string and set it to collapseWhitespace

and sent the results thru the json node and it works fine.

[E1cid]

Never used the string node, Would that collapse all space in the string values of the properties to? i.e would test test become testtest

[bonito]

@zenofmud

node-red-contrib-string

Hi E1cid

No I do not have much control it is a web relay with this interface. I will try your suggestion,
Thank you for helping.

[bonito]

Hi zenofmud,

Thank you very much for you help, it works perfectly with the string - white space remove.

It would never occur to me. It is difficult if I am trying to learn something new and it works differently than it should. Thank you very much for you help and time and patiance with me.

Best regards

[zenofmud]

From the 'Flows' page for the node

Additional string parsing functionality and compatibility have been added from the fork.

If you click on the fork link and read thru you will see:

- collapseWhitespace()

Converts all adjacent whitespace characters to a single space.