Python Script into .JSON file for node red?

No it didn't break i stopped it :slight_smile:
Ok I'll delete them. sorry for the late response, I had to work in another town today.

No problem,
Good to hear, the script is working
You see the data and there are examples in how to publish via mqtt
I guess you know how to get the parts you need and format a message that you can send via mqtt
I recommend that you create a JSON structure and send that to NR
In python you can use import json and json.dumps

EDIT: A better example
Assume you have calculated the MP2,5 and MP10 values based on the received data. In python you can easily construct a JSON structure. This will be a fine format to send to NR where it then will be very easy to take those values and present them further

So in the script, add on top
import json

Then in the the code where you have the data from the sensor, assume you do the needed calculation and the result might end up with something like below. Use json.dumps to make the result as a string that can be transferred via MQTT and then publish it

res = {"MP2.5":5.8, "MP10":7.2}
res = json.dumps(res)
result, mid = client.publish("sds011/data", res, 0)

In NR the result of the above will look like below. If you add a JSON node after the MQTT in node, you get the result as a javascript object (actually you can set this already in the configuration of the MQTT in node, then skipping the JSON node)

Thank you very much !

Where do i see the data in NR ? I see them only as an array in the command prompt.
And what i forgot to say, is that i can start the script only in the command prompt, not in NR. When i press "Start script" nothing happens, only when i start it in the command prompt with "sudo python3 sds011.py".

And i think i understand your example(a little bit), but I already prepared function nodes to calculate with the PM2,5 & PM10 values. The point is to calculate an Air Quality Index (AQI). I already calculated an AQI with temperature, humidity and CO2 concentration. Than I organized them into numbers from 1-5, and after this I organize all of the 4 AQI's to output the baddest one with Math.max().
Here is an example. And everything works. So i just need to get the PM2,5 & PM10 values in the output.



But because this is all extremly new to me, I really don't know how to get only them as an output.
Do I have to make the same steps you told in your example or is it completely different ?
Best wishes

Well. if you did not follow my example, you do not. Thats why I made the example there to show you how to publish data to NR via MQTT. But first you need to format the data in a useful way so you can grab it easily in NR and do your calculations. I thought you had already done that by now

Did you modify the command in the exec node? Maybe you need full path to the script and also add sudo in front. And maybe your display iss not 0, maybe try with DISPLAY:1

The PM2,5 and PM10 values seems to come as a low and high byte respectively. Do you say you would like to have them as they are? Like in one of the lines you showed earlier, the PM2,5 value would be data[2] and data[3], as example 145 and 0. Is that the values you would like to be sent to NR for PM2,5? And the eqvivalent for PM10?. The best seems to send data in a format as I described so that you get a javascript object holding the data you need

Yes, as discussed above but it is not too difficult

Just to show how you can do it

If you modify the code in the python script as follows, it will be simple to get the values for your calculation in NR

In the beginning of the script, add
import json

Then modify the code in the while loop:

    while data[0] != 3 and not th_abort:
        try:
            data = device.read(endpoint.bEndpointAddress, endpoint.wMaxPacketSize)
            print(data)

            res = {"MP2_5LB":data[2], "MP2_5HB":data[3], "MP10LB":data[4], "MP10HB":data[5]}
            res = json.dumps(res)
            result, mid = client.publish("sds011/data", res, 0) 

            sds011_thread_Event.wait(1)
        except usb.core.USBError as e:
            if e.args == ('Operation timed out',):
                print("timeout)")
                continue

Depending on how often you need to read the data from the sensor, you can change this in the
sds011_thread_Event.wait(1)

To read every 5 second, sds011_thread_Event.wait(5)

In NR you add code in your function node to get the values like this example

let MP2_5LB = msg.payload.MP2_5LB;
let MP2_5HB = msg.payload.MP2_5HB;
let MP10LB = msg.payload.MP10LB;
let MP10HB = msg.payload.MP10HB;

If you need other parts of the data like id and checksum you can easily extend the example

Thank you very much!
I added the code and everything worked :slight_smile: also in NR the data showed.
But now i get an mqtt error, do you know where the mistake can be ? :smiley:

EDIT:

I don't know what I've done but now, they're isn't data anymore in the debug window :smiley: I changed nothing in the script. Just getting the following.
I also put a debug node to control that only the MP_2_5HB and MP10HB values come out.


Screenshot 2021-05-17 101816

Hmm, don't really know but could it be you have some old threads still hanging around there?
I would like to make a new version of the script and remove those sys.exit things and handle it differently if the usb connection fails. Also, terminate the script using the command in NR instead of Ctrl+C

Updated in a few minutes...

The situation now is the following:

After i modified the script,I started it and everything worked, also with the "MP2_5HB = 92" etc. in the debug window. Shortly after I had this mqtt error. Than i checked everything, reboot the Pi and made an update. Now if i start the script from NR, nothing shows in the debug window, but the command prompt opens automatically und the program runs. If I start the script directly from the command prompt, the debug window shows the object, but without the modification, wich i can not explain :smiley: I changed nothing in the script after the modification, but it only worked once after I started it the first time.

Here is the modification:

Let's take one thing at the time

Try the script by starting it from the command prompt and kill it by sending the abort command from NR
(we will look at the problem when starting it NR when we see the script is running without problems)

I have modified the script a bit so testing that the usb connection with the sensor is operational, step by step before we start to read data. Let's see how it works

sds011.txt (3.1 KB)

EDIT: Just to mention, you have connected the function node to the wrong mqtt in node. The data arrives at sds011/data

ooooooh sorry, my bad. I connected it correct now and everything works. The script starts from NR and the debug window shows the modified code :slight_smile: :smiley: I knew I messed up something :smiley:

I get the following

I tried, but I didn't managed to seperate the payload in the function node, that i have two values to calculate with.
The function node is still the following.

Do I have to run your new script to solve this problem or do I have to make other changes ?

Thank you.

Use my new script to allow me to support you much better, it has improvements I think (keep a copy of the old script or just rename it as old)

For the code in your function node, try

let MP2_5LB = msg.payload.MP2_5LB;
let MP10LB = msg.payload.MP10LB;

return [
    {payload:MP2_5LB},
    {payload:MP10LB}
    ];

Thank you.

I now use your new script and everything works fine :slight_smile:
But still, i get the payload as undefined.

Good the script works!
Can you show the output from the command prompt so I can see what the script writes?
Can you change the setting of the mqtt node to provide "a parsed JSON object" ?
Can you change the setting for the debug node to show the complete msg?

Now It works !!!


The debug window says on top "failed to analyse the JSON-String"

Right, my mistake, you can comment out this line
result, mid = client.publish("sds011/data", "Starting data reading", 0)
or change the topic to another one if you want to keep the info sent

Good result I think! Enjoy the sensor readings for a fresh air!

Thank you very very much for all your help! I have two last questions.

  1. Now, the data comes every second, can i somehow change the time?
  2. And at last, can i use this script and nodes (with the associated VID, PID and new mqtt adress) for other sensors?

But for now, I don't now how i can thank you for all of your help and patience. :smiley: :slight_smile:

1 Like

I'm happy it works for you!! I think you did great, it is an interesting exercise and it shows how well a Python script can work and communicate with Node-RED via MQTT. Myself I have many such scripts using MQTT for various needs in my own system. They have been working just excellent since years. One thing we did not discuss so far is the potential using MQTT. Imagine you have a sensor in a remote place that you would like to monitor. If you use MQTT in the same way as we just did, you can instead use a MQTT broker service in the cloud. There are several such service providers offering a secure connection for MQTT, very popular in IoT applications

My intention is not to divert from the topic but another very possible way of connecting and collecting data from sensors is to use a small ESP device. I have a couple of those ESP32's and it is dead simple to make it happen. You can check out this site where I actually found a solution supporting sds011 including communication via MQTT. So that could also be a way and in that case, no Python script needed. As you can see they have solutions ready for many sensors of different kind

Now to answer your questions

  1. Yes, you just change one thing in the script in line 53 (in the latest script):
    sds011_thread_Event.wait(1)
    for data every ten second, change to
    sds011_thread_Event.wait(10)
    or any other interval you prefer (just insert number of seconds to wait in between)

  2. Well, depends. If the sensors are compatible then it would be possible, if they communicate in the same way with the same "protocol". If the data they deliver looks different and is in different positions, the script would need to be modified. But the principle should work; having one thread reading the data and one thread for the mqtt communication

Best regards, Walter

1 Like

Thank you for all your tips! This project now is just from my university, but after all the experience I made I really gained interest and try to set some things up at my place :smiley:

I really don't want to bother you anymore but I have some new little problems with the function nodes i calculate with, do you have time to look at it ?

Best wishes

Go ahead...