More fun with message formatting

Ok, to keep things separate:

I'm stuck now with a message I have made with a python3 script to make a JSON message.

The message as given is:

{"ssid": "Marys_Farm_2.4", "signal_level": "-72"}
{"ssid": "PiNet", "signal_level": "-70"}
{"ssid": "Optus_ABA172_5GHz", "signal_level": "-48"}
{"ssid": "Marys_Farm_5", "signal_level": "-55"}
{"ssid": "WiFi-AFB6", "signal_level": "-49"}
{"ssid": "TelstraCA5E31", "signal_level": "-64"}
{"ssid": "MyRepublic 49D9", "signal_level": "-63"}
{"ssid": "Optus_ABA172", "signal_level": "-76"}
{"ssid": "WiFi-F852-5G", "signal_level": "-68"}
{"ssid": "Aussie Broadband 4928", "signal_level": "-49"}
{"ssid": "\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00", "signal_level": "-58"}
{"ssid": "WiFi-F852", "signal_level": "-74"}
{"ssid": "WiFi-AFB6-5G", "signal_level": "-80"}
{"ssid": "WiFi-E987", "signal_level": "-72"}
{"ssid": "OPTUS_69E639M", "signal_level": "-82"}
{"ssid": "TelstraDDE916", "signal_level": "-85"}
{"ssid": "Not Your Wifi_Guest", "signal_level": "-84"}

And I am getting an error:

{"message":"Unexpected token { in JSON at position 50","source":{"id":"177e50e0152c6adc","type":"json","name":"WAP LIST JSON","count":1}}

Which after a bit of digging is (depending if the value is ZERO counting or not:
The end of the first line of the start of the second.

As the error declared { I'm guessing the start of the second line.
Ok, so (again guessing) there is a , missing between the first { } and the second one.

I could maybe get the python code to stick a , at the end of the line but then when it gets to the end and sends the LAST line, a , would be stuck at the end of that, and I am sure that will throw a spanner in the works.

SORRY... I know this is python and this forum is Node Red.
But I feel here I am better known and the answer/s may be better worded for me than if I am a complete noob in a python forum and ask the same question.

Your guess is correct. You need a comma between each object. AND you also need a [ at the very start and a ] at the very end of the last line in order to create a complete json array object that can then be parsed.


Alas that is above my skill set.


I'll play around with it.

The first/last [ ] could be done with a template node - yes?

It would be better to do it correctly in python to start with.

1 Like

Yes, agreed. But as I said: That is a beyond vertical curve just now.

I'll run off and see what I can learn ITMT.



Searching and trying to understand.

Found this link Write json into file Python 3 - Stack Overflow

But in those examples the JSON parts are not separated with a ,

They are just appended to one another - as I kind of have. (Ok, mine have a \n between them)

So I am missing something in the structure I fear/feel.

Still looking - kind of.

Ideally you would only send valid json. However, of this is all you have then in a function...

msg.payload = JSON.parse(`[ ${msg.payload.split('\n').join(',')} ]`)
return msg


1 Like

(Sorry for late/slow reply)

That's in a function node in NR - yes?

Just as DCJ said: Better to keep it all in python.


As did I...

But I gave you an (untested) workaround

1 Like

I am not having a good time just now.

My parsing system is not as good as I would like.

I'll see what happens.

But initial test gives me:

SyntaxError: Unexpected token , in JSON at position 3

I'll work on that and see what I can work out.

Change it to this:

const json = `[ ${msg.payload.split('\n').join(',')} ]`
msg.payload = JSON.parse(json)
return msg


What do you see?

1 Like

Just before I do that:
(Not having a go at you)

I stuck some debug nodes in the code.

Going in is one big message.
After that node it has split them into single messages.

(See below)

Going in:

{"ssid": "Marys_Farm_2.4", "signal_level": "-49"}
{"ssid": "Marys_Farm_5", "signal_level": "-53"}
{"ssid": "TelstraCA5E31", "signal_level": "-26"}
{"ssid": "PiNet", "signal_level": "-70"}
{"ssid": "Optus_ABA172_5GHz", "signal_level": "-80"}
{"ssid": "WiFi-E987", "signal_level": "-53"}
{"ssid": "\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00", "signal_level": "-48"}
{"ssid": "WiFi-AFB6", "signal_level": "-51"}
{"ssid": "Optus_ABA172", "signal_level": "-59"}
{"ssid": "WiFi-F852", "signal_level": "-64"}
{"ssid": "MyRepublic 49D9", "signal_level": "-75"}
{"ssid": "WiFi-AFB6-5G", "signal_level": "-85"}
{"ssid": "Madagascar", "signal_level": "-68"}
{"ssid": "Aussie Broadband 4928", "signal_level": "-87"}
{"ssid": "TelstraDDE916", "signal_level": "-66"}
{"ssid": "OPTUS_69E639M", "signal_level": "-83"}
{"ssid": "NetComm 4761", "signal_level": "-80"}

Coming out:

See picture.

I am suspicious the join part is failing somehow/where.


This is what I get:

[ {"ssid": "Marys_Farm_2.4", "signal_level": "-48"},{"ssid": "Marys_Farm_5", "signal_level": "-53"},{"ssid": "TelstraCA5E31", "signal_level": "-21"},{"ssid": "PiNet", "signal_level": "-69"},{"ssid": "Optus_ABA172_5GHz", "signal_level": "-53"},{"ssid": "\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00", "signal_level": "-48"},{"ssid": "WiFi-AFB6", "signal_level": "-51"},{"ssid": "Optus_ABA172", "signal_level": "-68"},{"ssid": "MyRepublic 49D9", "signal_level": "-74"},{"ssid": "WiFi-AFB6-5G", "signal_level": "-67"},{"ssid": "Aussie Broadband 4928", "signal_level": "-60"},{"ssid": "WiFi-F852", "signal_level": "-64"},{"ssid": "WiFi-F852", "signal_level": "-80"} ]

But it isn't JSON format. :frowning: (I think)
The output - when sent to the chart - does nothing/nothing is shown.

It is valid json.

Check for yourself:

Actually, if you look at your debug, you can see it has been parsed to a js array of objects. But then you run it through JSON node again - that turns it back into a string.

In short, delete the JSON node.

Sorry Steve. I do accept your reply as valid.

But I only added the second JSON node because the debug node wasn't showing it in the style I understand as valid JSON.

(Hang on while I test it)

This is what I get/see.

But if I go to the chart node.....


I'll stick the spare debug node after the change node.
Hang on. (again) :slight_smile:

This is what I get:



[{"id":"5857cdacd0e8c87a","type":"function","z":"2eb377ce0b03a7d9","name":"function 9","func":"const json = `[ ${msg.payload.split('\\n').join(',')} ]`\nnode.warn({ json })\nmsg.payload = JSON.parse(json)\nreturn msg","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":240,"y":300,"wires":[["6bbb8da9ee9f8f2a","04a6910f87afe4ca","177e50e0152c6adc"]]},{"id":"177e50e0152c6adc","type":"json","z":"2eb377ce0b03a7d9","name":"WAP LIST JSON","property":"payload","action":"","pretty":false,"x":450,"y":300,"wires":[["48ee756e1ec0c783","f09abc35ef453f9c"]]},{"id":"f09abc35ef453f9c","type":"change","z":"2eb377ce0b03a7d9","name":"","rules":[{"t":"move","p":"payload.ssid","pt":"msg","to":"topic","tot":"msg"},{"t":"move","p":"payload.signal_level","pt":"msg","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":730,"y":330,"wires":[["a5f43014722527f0","153c6bc8ba3c1a80"]]},{"id":"a5f43014722527f0","type":"ui_chart","z":"2eb377ce0b03a7d9","name":"","group":"bcfab13aa8c1ed64","order":0,"width":"8","height":"10","label":"chart","chartType":"line","legend":"true","xformat":"HH:mm:ss","interpolate":"linear","nodata":"","dot":false,"ymin":"","ymax":"","removeOlder":1,"removeOlderPoints":"30","removeOlderUnit":"3600","cutout":0,"useOneColor":false,"useUTC":false,"colors":["#1f77b4","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"outputs":1,"x":880,"y":300,"wires":[[]]},{"id":"bcfab13aa8c1ed64","type":"ui_group","name":"Group 1","tab":"1bc896de46b4f98a","order":1,"disp":true,"width":"10","collapse":false},{"id":"1bc896de46b4f98a","type":"ui_tab","name":"WAP list","icon":"dashboard","order":10,"disabled":false,"hidden":false}]

That I am getting undefined from that debug node implies to me that the structure is not valid in the message.


As (I think) everyone has agreed, it is best to fix any message format problems at source, so I have taken another look at the Python script you posted in the other thread.
Here is the script as I modified it, with comments to explain the change.
Essentially I'm making a Python dictionary for each access point then adding it to an array.

#! /usr/bin/python
import json
import subprocess
import re

def scan_wifi():
    # Run the iwlist command to scan for wireless networks
    cmd = subprocess.Popen(['sudo', 'iwlist', 'wlan0', 'scan'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    output, _ = cmd.communicate()

    # Decode the output from bytes to string
    output = output.decode()

    # Extract the relevant information using regular expressions
    networks = re.findall(r'ESSID:"(.*?)".*?Frequency:(.*?)\s.*?Signal level=(.*?)\s', output, re.DOTALL)

    # Return the list of networks
    return networks

wifi_networks = scan_wifi()
myarray = []    # The output should be an array of js objects, so make an empty array
for network in wifi_networks:
    ssid = network[0]
    frequency = network[1]
    signal_level = float(network[2])       # Take the opportunity to have signal level as a number not string
    dict = {"ssid": ssid, "signal_level": signal_level} # Make it a Python dictionary {name:value, name:value}
    myarray.append(dict)  # Add it to the array

output = json.dumps(myarray) # Finally output as json string
print (output)

This is the slightly modified flow to call the script and process it

[{"id":"77b306137df5482d","type":"ui_chart","z":"4f02d48c1fd987a5","name":"","group":"04d86d5dc7122996","order":2,"width":0,"height":0,"label":"chart","chartType":"line","legend":"true","xformat":"HH:mm:ss","interpolate":"bezier","nodata":"","dot":false,"ymin":"","ymax":"","removeOlder":1,"removeOlderPoints":"","removeOlderUnit":"3600","cutout":0,"useOneColor":false,"useUTC":false,"colors":["#1f77b4","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"outputs":1,"useDifferentColor":false,"className":"","x":890,"y":500,"wires":[[]]},{"id":"f11072b8baee6804","type":"exec","z":"4f02d48c1fd987a5","command":"/home/pi/foo","addpay":"","append":"","useSpawn":"false","timer":"","winHide":false,"oldrc":false,"name":"","x":270,"y":420,"wires":[["7a6ec66b4a5acd27"],[],[]]},{"id":"867be5fe367ced2f","type":"inject","z":"4f02d48c1fd987a5","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"30","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":110,"y":420,"wires":[["f11072b8baee6804"]]},{"id":"a33eafdd94fae24a","type":"inject","z":"4f02d48c1fd987a5","name":"empty array to clear chart","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"[]","payloadType":"json","x":150,"y":500,"wires":[["77b306137df5482d"]]},{"id":"7a6ec66b4a5acd27","type":"json","z":"4f02d48c1fd987a5","name":"","property":"payload","action":"","pretty":false,"x":410,"y":420,"wires":[["3f36a9d814642f40"]]},{"id":"3f36a9d814642f40","type":"split","z":"4f02d48c1fd987a5","name":"Split on comma","splt":",","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":560,"y":420,"wires":[["fefa2028f47708e6"]]},{"id":"fefa2028f47708e6","type":"change","z":"4f02d48c1fd987a5","name":"payload, topic","rules":[{"t":"move","p":"payload.ssid","pt":"msg","to":"topic","tot":"msg"},{"t":"move","p":"payload.signal_level","pt":"msg","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":740,"y":420,"wires":[["048800b73d1a39f5"]]},{"id":"048800b73d1a39f5","type":"switch","z":"4f02d48c1fd987a5","name":"","property":"topic","propertyType":"msg","rules":[{"t":"cont","v":"TALKTALK","vt":"str"},{"t":"cont","v":"BT-","vt":"str"}],"checkall":"true","repair":false,"outputs":2,"x":890,"y":420,"wires":[["77b306137df5482d"],["77b306137df5482d"]]},{"id":"04d86d5dc7122996","type":"ui_group","name":"Default","tab":"f3a8a67ff0904e39","order":2,"disp":true,"width":"6","collapse":false,"className":""},{"id":"f3a8a67ff0904e39","type":"ui_tab","name":"HOME","icon":"dashboard","order":2,"disabled":false,"hidden":false}]

My script is /home/pi/foo. Not a wise choice of name but it's experimental. And I'm calling it via the exec node. I think you are using a pythonshell node. I assume it's much the same.

I have to pass the returned string through a json node because exec only returns a string.

I split on commas not new lines because it's now a proper json object.

Setup payload and topic for the chart same as you.

I'm using a switch node to filter out just a few SSIDs for clarity.
Note that there are some accesspoints with blank or weird SSIDs which really mess up the chart, so this switch plays an important role.

Here is an example of the chart


I love it. @jbudd

Thanks for the effort/time you put in.

Yes, I am slightly suspicious of some of the WAP names I see. But that's life.

That is really good.

How did you get the round lines on the chart? (bezier?)
I am on NR 3.0.2

That's an easy one to answer - select 'bezier' on the Interpolate option on the Chart node.

1 Like


I've never used that before.

(OH and sorry.....)

Can I add more colours to the chart node?
9 isn't quite cutting it.

There are 24 (?) more colours specified somewhere in the background.

Sorry, but when I open the chart node I only see 9 colours.

(I'm brain dead just now).