MQTT data to a mySQL to display data on public website

Hi folks,

Firstly I'm not a programmer but can follow instructions quite well.
I have an ESP32 sending sensors data to Node-red on a Raspberry Pi, acting as an MQTT broker. The MQTT broker subscribes to a single topic that outputs the data as "a
parsed JSON object." The debug look like this:

{"controllertype":"mySQM+","firmware":127,"uptime":"00:09:04","date":"01/01/1900","time":"00:00:00","sqm":17.74,"nelm":3.75,"lux":0.0087,"ambient":23.67,"humidity":57.2,"dewpoint":14.71,"pressure":1014.22,"slpressure":1015.06,"bme280alt":7,"skyambient":20,"skyobject":20,"skystate":1,"cloudcover":100,"raining":0,"rvout":0,"rainprevhr":0,"raincurrhr":0,"raincurrday":0,"windspd":0,"windavg":0,"windgust":0,"windchill":100,"beaufort":0,"winddir":0,"gpsdate":"01/01/1900","gpstime":"00:00:00","gpslat":"-77 00 00 S","gpslon":"166 00 00 E","gpsalt":0,"gpssat":0,"gpsfix":0,"mac":"78:E3:6D:0A:24:14","makehay":12.52}

On the domain, the MySQL table will look something like the snippet below which should only contain "ambient", "humidity" and "pressure" data.

CREATE TABLE Sensor (
    id INT(6) UNSIGNED AUTO_INCREMENT PRIMARY KEY,
    value1 VARCHAR(10),
    value2 VARCHAR(10),
    value3 VARCHAR(10),
    reading_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
)

From Node-red I would like to send the data to the MySQL database using a PHP post script fround on the website which looks like this example:

$servername = "localhost";

// REPLACE with your Database name
$dbname = "REPLACE_WITH_YOUR_DATABASE_NAME";
// REPLACE with Database user
$username = "REPLACE_WITH_YOUR_USERNAME";
// REPLACE with Database user password
$password = "REPLACE_WITH_YOUR_PASSWORD";

// Keep this API Key value to be compatible with the ESP32 code provided in the project page. If you change this value, the ESP32 sketch needs to match
$api_key_value = "tPmAT5Ab3j7F9";

$api_key = $value1 = $value2 = $value3 = "";

if ($_SERVER["REQUEST_METHOD"] == "POST") {
    $api_key = test_input($_POST["api_key"]);
    if($api_key == $api_key_value) {
        $value1 = test_input($_POST["value1"]);
        $value2 = test_input($_POST["value2"]);
        $value3 = test_input($_POST["value3"]);
        
        // Create connection
        $conn = new mysqli($servername, $username, $password, $dbname);
        // Check connection
        if ($conn->connect_error) {
            die("Connection failed: " . $conn->connect_error);
        } 
        
        $sql = "INSERT INTO Sensor (value1, value2, value3)
        VALUES ('" . $value1 . "', '" . $value2 . "', '" . $value3 . "')";
        
        if ($conn->query($sql) === TRUE) {
            echo "New record created successfully";
        } 
        else {
            echo "Error: " . $sql . "<br>" . $conn->error;
        }
        $conn->close();
    }
    else {
        echo "Wrong API Key provided.";
    }
}
else {
    echo "No data posted with HTTP POST.";
}
function test_input($data) {
    $data = trim($data);
    $data = stripslashes($data);
    $data = htmlspecialchars($data);
    return $data;
}

I'm not sure how to do this in Node-red, any guidance would be much appreciated.

Jairo

I doubt you have to use a php script to enter data to your MYSQL DB, there are plenty of MYSQL nodes available to enter your data
e.g. Library - Node-RED.

Using the first from the list, here is an low code example of how to create the query in msg.topic

[{"id":"2b1eedce.9a031a","type":"inject","z":"bf9e1e33.030598","name":"incoming MQTT","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"controllertype\":\"mySQM+\",\"firmware\":127,\"uptime\":\"00:09:04\",\"date\":\"01/01/1900\",\"time\":\"00:00:00\",\"sqm\":17.74,\"nelm\":3.75,\"lux\":0.0087,\"ambient\":23.67,\"humidity\":57.2,\"dewpoint\":14.71,\"pressure\":1014.22,\"slpressure\":1015.06,\"bme280alt\":7,\"skyambient\":20,\"skyobject\":20,\"skystate\":1,\"cloudcover\":100,\"raining\":0,\"rvout\":0,\"rainprevhr\":0,\"raincurrhr\":0,\"raincurrday\":0,\"windspd\":0,\"windavg\":0,\"windgust\":0,\"windchill\":100,\"beaufort\":0,\"winddir\":0,\"gpsdate\":\"01/01/1900\",\"gpstime\":\"00:00:00\",\"gpslat\":\"-77 00 00 S\",\"gpslon\":\"166 00 00 E\",\"gpsalt\":0,\"gpssat\":0,\"gpsfix\":0,\"mac\":\"78:E3:6D:0A:24:14\",\"makehay\":12.52}","payloadType":"json","x":170,"y":140,"wires":[["f57147da.fb29"]]},{"id":"f57147da.fb29","type":"template","z":"bf9e1e33.030598","name":"","field":"topic","fieldType":"msg","format":"handlebars","syntax":"mustache","template":"INSERT INTO Sensor (value1, value2, value3) VALUES ('{{payload.ambient}}', '{{payload.humidity}}', '{{payload.pressure}}')","output":"str","x":370,"y":140,"wires":[["709bd9d3.7011e8"]]},{"id":"709bd9d3.7011e8","type":"debug","z":"bf9e1e33.030598","name":"MYSQL query","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"topic","targetType":"msg","statusVal":"","statusType":"auto","x":570,"y":140,"wires":[]}]

This will enter your data as strings, you maybe better to change your DB table to have your readings as numeric(floats/decimals).
[edit] Fixed typo cheers @jbudd

1 Like

Personally, I would use this one as it well supported.

2 Likes

I agree, use node-red-node-mysql to write to the database, it's very reliable and much simpler than calling an external script.

I think @E1cid's example script has a typo - `{{payload.pressure}}' should be '{{payload.pressure}}' ? (Different quote character)

Hi @E1cid ,
Thanks for the quick response.Some additional questions.
If I understood correctly, I would need to insert a 'change' node after the MQTT node to prepare the msg.payload which will contain the query setup named msg.topic, right? After that, the "MySQL" node will send data to the DB as shown in the screenshot below. Does that make sense?
Screenshot 2022-03-20 130255

The "change" node shall looked something like this?
Screenshot 2022-03-20 130524

In the MySQL node the database setting should look something like this? Would it connect to my host without needing port forwarding?

As for the MySQL DB table, this should work, right?

CREATE TABLE Sensor (
    id INT(6) UNSIGNED AUTO_INCREMENT PRIMARY KEY,
    value1 FLOAT(3,2),
    value2 FLOAT(3,2),
    value3 FLOAT(3,2),
    reading_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
)

Thanks in advance for the support.

Jairo

Yes it should connect without port forwarding.

The inject node i supplied was just for testing as i do not have access to your mqtt topic paayload.

you would connect the mqtt node to the template node, then the template to the mySQL node.
The template node would not need the single quotes if you are db now accepts floats.

INSERT INTO Sensor (value1, value2, value3) VALUES ({{payload.ambient}}, {{payload.humidity}}, {{payload.pressure}})

The template node outputs the query which is in the message property msg.topic, this is passed to the mySQL node.

1 Like

This:

Is not correct!

The code that E1cid posted is a complete flow. Import it to Node-Red using the hamburger menu.

It seems a bit odd to use "value1" etc as database field names, why not "ambient", "humidity", "pressure"?

2 Likes

@E1cid thanks again for the quick reply.
And thanks for clarifying the template part. I have now set up the flow like this.
Screenshot 2022-03-20 142445

However, I'm receiving some errors connecting to the database. Here's the screenshot and settings in the MySQL node.

In phpMyAdmin on the domain, the host appears to be localhost:3306 this should relay to https://skynstars.com, right?

Screenshot 2022-03-20 143439

Where is you DB set up is it on same machine or hosted on the web hosted server?

It's on the web-hosted server.

Then i think you will have to configure the server to allow remote access to mySQL. Long time since i hosted on web. You may have to ask the hosting company how to achieve this.

Ok, here's a solution I found to inject values into the MySQL DB on the web-hosted server.
I've used the HTTP Request node (POST) that calls in an API key found in a PHP script on the web-hosted server (e.g., URL: http://example.com/weather/post-data.php). Once the API key is verified, the PHP script injects the values into the DB. Not sure if this is a safe way to put values in the DB, but as shown on the screenshot below, it's creating the table. What are your thoughts on this?

Here's what the flow looks like:

Here's the change node setting:

The data being injected into the DB:

The PHP script used on the web-hosted server to inject values in the DB looks like this:

<?php

$servername = "localhost";

// REPLACE with your Database name
$dbname = "REPLACE_WITH_YOUR_DATABASE_NAME";
// REPLACE with Database user
$username = "REPLACE_WITH_YOUR_USERNAME";
// REPLACE with Database user password
$password = "REPLACE_WITH_YOUR_PASSWORD";

// Keep this API Key value to be compatible with the ESP32 code provided on the project page. If you change this value, the ESP32 sketch needs to match
$api_key_value = "tPmAT5Ab3j7F9";

$api_key = $value1 = $value2 = $value3 = $value4 = $value5 = $value6 = "";

if ($_SERVER["REQUEST_METHOD"] == "POST") {
    $api_key = test_input($_POST["api_key"]);
    if($api_key == $api_key_value) {
        $value1 = test_input($_POST["value1"]);
        $value2 = test_input($_POST["value2"]);
        $value3 = test_input($_POST["value3"]);
        $value4 = test_input($_POST["value4"]);
        $value5 = test_input($_POST["value5"]);
        $value6 = test_input($_POST["value6"]);
        
        // Create connection
        $conn = new mysqli($servername, $username, $password, $dbname);
        // Check connection
        if ($conn->connect_error) {
            die("Connection failed: " . $conn->connect_error);
        } 
        
        $sql = "INSERT INTO Sensor (value1, value2, value3, value4, value5, value6)
        VALUES ('" . $value1 . "', '" . $value2 . "', '" . $value3 . "', '" . $value4 . "', '" . $value5 . "', '" . $value6 . "')";
        
        if ($conn->query($sql) === TRUE) {
            echo "New record created successfully";
        } 
        else {
            echo "Error: " . $sql . "<br>" . $conn->error;
        }
    
        $conn->close();
    }
    else {
        echo "Wrong API Key provided.";
    }

}
else {
    echo "No data posted with HTTP POST.";
}

function test_input($data) {
    $data = trim($data);
    $data = stripslashes($data);
    $data = htmlspecialchars($data);
    return $data;
}

Using http over the web with your api key is not very secure, It would be best to use https

1 Like

Understood...

On a slightly different topic. With the same sensor values, I've created a node-red dashboard that I can only access on my local network. The screenshot can be found below.
Is there a method to display this screen on a public webpage?

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.