JSON node cannot parse NaN coming from Arduino Serial

General
1

I have an Arduino UNO connected via USB to RPi. The Arduino reads a Temperature-Humidity sensor, forms a JSON object with the ArduinoJson library, and sends it through serial to RPi, received by NodeRED. There I parse the JSON, and everything works fine, except when I disconnect the sensor from Arduino. The values read are nan, and thus the JSON formed carries a NaN instead of a value. This causes the JSON node to report a parsing error. Why? How do I have to handle NaN values?

I wonder if the only way is to filter the incoming JSON string manually with a function node. In this case I could the whole parsing there as well. So, is the JSON node useless?

Here is part of my arduino code:

#include <ArduinoJson.h>
#include <DHT.h>
#include <DHT_U.h>

float hum;
float temp;
StaticJsonBuffer<200> jsonBufferIn;

void loop() {
  hum = dht.readHumidity();
  temp= dht.readTemperature();

  StaticJsonBuffer<200> jsonBufferOut;
  JsonObject& root = jsonBufferOut.createObject();

  root["AirTemp"] = temp;
  root["Humidity"] = hum;
  
  root.printTo(Serial);
  delay(30000);
}
2

Use a switch node

23
Screenshot 2019-04-22 at 18.45.23.jpg1000Ă—656 46.3 KB

Data will only get through if it is a number. (can also check if it is an object)

3

NaN is not valid JSON so that is why it errors. So yes I’m afraid you need to pre-filter it

4

Using the switch node to filter is a good idea but it will not work with the configuration shown. The reason is that NaN is a special number value.

Better using the output 2 for a config like below.

r-01

Try

2 Likes
5

The reason is that NaN is a special number value.

ah yes you are correct.

6

Ops... I guess my filter is not working.... need to redo, probably using JavaScript isNan() or Number.isNaN() function.

Here is a post from the old forum that addressed a similar issue

7

Given you have control over the Arduino code, I'd suggest adding in appropriate code on that side so you don't ever send invalid JSON. For example, check for valid values and return a known value that won't occur normally that you can test for more easily in your NR flow.

1 Like
8

On the smaller (in terms of processing power) MCs the less processing of the data the better. Many work on a simple "collect-dispatch" methodology for sensor data, so NR should be the ideal platform to work on the raw data. No?

Even in this case I wouldn't be letting the Arduino do anything json. Just send through the raw data and let NR do the work.

9

If that is really an issue, you wouldn't be outputting JSON but instead a simpler structured text format.

Seriously, I've never had an issue outputting JSON even with quite complex, multi-sensor code on Arduino's and ESP8266's though I don't tend to use a JSON library since they can be fairly inefficient. Instead I simply use a function that contains fixed text that can be offloaded to FLASH using F("My text").

1 Like
10

The problem described here is: except when I disconnect the sensor
The sensor won't put out JSON when that happens, or it may give garbled data which needs to be handled somewhere else, ie. node-red.

11

If you don't want those values, why not filter them out at source. It isn't unusual, especially with the DHT's, to receive rubbish or no data - so don't send anything if the data is bad.

On my systems, I just catch the JSON node error and ignore it if I get a garbled message, only ever happens with an Arduino that is serial connected and you are right, it most often happens on disconnect and occasionally on a connect.

12

The only reason I use JSON, is to validate whether my data has been sent correctly. I've tried a lot of different (custom and not) protocols, but JSON is the easiest way to check if something has gone wrong during transfer. There's no sufficient way to check for corruption if I send unformatted raw data. And you are right about the efficiency, so that's why I want to catch the NaN values on NR. If {"key": NaN} is not a valid JSON, then I don't understand why ArduinoJson creates it this way.

13

@bakman2 and @TotallyInformation
When the DHT sensor is disconnected physically from the arduino, trying to read the sensor gives back a nan value. When putting that in a JSON string with ArduinoJson becomes NaN.

I want this information passed on the RPi(NR), so I can create notifications that something is wrong with the sensor.

I could do the checking on arduino and create a report JSON e.g. {"error": "DHT"}, however I want to avoid doing much processing on arduino, and instead detect on NR that I get NaN instead of a value, and handle from there on.
E.g. instead of {"AirTemp": 23, "Humidity": 32}, I get {"AirTemp": NaN, "Humidity": NaN}.

If those NaN values were strings, I guess there would be no problem. Not sure why ArduinoJson puts the NaN in this way.

14

Indeed. For the record though, the "old fashioned" way of doing this is to add a checksum to each returned record. Something as simple as a character count works just as well if the line lengths are reasonably predictable. But yes, JSON is fine and I took to using it a long time back - as mentioned I simply drop anything that gets corrupted in-transit - of course, that relies on their being another record of the same type being sent in not too long a time-frame. This is absolutely fine for sensor platforms.

Just filter it out at the Arduino end - another reading will be along shortly. I don't know how rapidly you are reading the sensors, I read mine about once per minute so missing a reading gives two minutes between updates - still far more than I really need. I could easily halve that and still have a perfectly fine system.

If you get repeated NaN readings, set the output to a number that you know cannot be true. I tend to use -99. Easily good enough unless you are working with cryogenics - and then the DHT's wouldn't be much use :slight_smile: In truth, the DHT sensors are rubbish and I stopped using them a while back when alternatives dropped in price. I use digital sensors now like the BME/BMP 280 which are a lot more accurate, recover better from extreme readings, don't drift over time and are more responsive.

Honestly, unless you are doing some other really heavy processing on the Arduino, even a Nano has more than enough umph to do this kind of simple processing and it saves all this faffing around.

If you really want to keep the NaN values, you will have to surround them with double-quotes either on the Arduino or in your flow before it hits the JSON node. In my view, that is more work and is less robust. Do it on the Arduino in a function so that you can easily ditch it if you ever upgrade the sensor.

Because it is wrong and you might want to raise an issue against it. Or ditch it and set up the JSON manually, not hard to do with a simple function and a lot lighter on Arduino resources - giving you more than enough back to do the simple processing that I've outlined.

2 Likes
15

I don't think ArduinoJson is responsible, it sends data while you disconnect it, halve data packets arrive at node-red who tries to interpret, fails: NaN. Catch it at node-red.

16

Indeed, such data manipulation produces the desired outcome. JSON node will not complain when parsing the modified string. Here the testing flow on Node-RED.

[{"id":"bc36c4da.ab0df8","type":"tab","label":"Flow 7","disabled":false,"info":""},{"id":"cc570a91.ee27e8","type":"debug","z":"bc36c4da.ab0df8","name":"Debug3","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":660,"y":80,"wires":[]},{"id":"dac3ef75.55b15","type":"inject","z":"bc36c4da.ab0df8","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":140,"y":80,"wires":[["4ab230b9.030a1"]]},{"id":"4ab230b9.030a1","type":"function","z":"bc36c4da.ab0df8","name":"Replace NaN","func":"let str= \"{\\\"AirTemp\\\":NaN,\\\"Humidity\\\":10}\";\nnode.warn(str);\n\nmsg.payload = str.replace(\"NaN\", '\"NaN\"');\nnode.warn(msg.payload);\n\nreturn msg;","outputs":1,"noerr":0,"x":330,"y":80,"wires":[["33c2349a.f1aeec"]]},{"id":"33c2349a.f1aeec","type":"json","z":"bc36c4da.ab0df8","name":"","property":"payload","action":"obj","pretty":false,"x":490,"y":80,"wires":[["cc570a91.ee27e8"]]}]

s-01

17

You are right, I will create an issue. According to json.org the only acceptable unquoted value for a missing value would be null.

I have resolved my issue with a small workaround (see code below). I figured out how to pass null or the read value of the sensor unquoted. I pass them as String objects to the serialized() method. I hope they will fix it in the future, so I can skip that. My code differs a bit from the original post, because I have updated ArduinoJson in a hope to have it fixed in v6 (it wasn't, thus my workaround).

#include <ArduinoJson.h>
#include <DHT.h>
#include <DHT_U.h>

float hum;
float temp;

void loop() {
  hum = dht.readHumidity();
  temp= dht.readTemperature();

  StaticJsonDocument<200> root;

  root["AirTemp"] = isnan(temp) ? serialized(String("null")) : serialized(String(temp));
  root["Humidity"] = isnan(hum) ? serialized(String("null")) : serialized(String(hum));

  serializeJson(root, Serial);
  delay(30000);
}
2 Likes
18

This is the code I used in my sketches in the loop:

  // Reading temperature or humidity takes about 250 milliseconds!
  // Sensor readings may also be up to 2 seconds 'old' (its a very slow sensor)
  float h = dht.readHumidity();
  // Read temperature as Celsius (the default)
  float t = dht.readTemperature();
  // Read temperature as Fahrenheit (isFahrenheit = true)
  float f = dht.readTemperature(true);

  // Check if any reads failed and exit early (to try again).
  if (isnan(h) || isnan(t) || isnan(f)) {
    Serial.println("Failed to read from DHT sensor!");
    return;
  }

so if it doesn't get a reading it exits the loop and runs it again.

1 Like
19

Or switch to a better sensor :wink:

20

I literally pull the cable myself. I just want to prevent and error-proof any (well... as many) unexpected situations for the system I build :slight_smile: