How to parse MQTT payload object on ESP8266

Hello. I am sending MQTT data from my raspberry PI to the remote nodes ESP8266. In order to be efficient with the data and create as few topics as possible, I am sending a few different objects of information on a single topic as such:
My function node looks like that:


msg.topic = msg.device + '/item_inside'

msg.payload = {}; //create and empy object

msg.payload.item = msg.item

msg.payload.serial = msg.serial

msg.payload.quantity= msg.quantity
return msg

With this topic and payload information, I publish to an MQTT topic and send item, serial and quantity data.

On the ESP8266 side, I am using an example project and the data parsing function looks like that:

void callback(char* topic, byte* payload, unsigned int length) {
  Serial.print("Message arrived [");
  Serial.print(topic);
  Serial.print("] ");

  char message[length + 1];
  strncpy (message, (char*)payload, length);
  message[length] = '\0';
  Serial.println(message);

Serial monitor on my ESP8266 when I send data from raspberry PI

Message arrived [device1/item_inside] {"item":"item1","serial":"1A2B","quantity":50}

Also, on my ESP8266 code, I have a structure:

struct item_inside_struct {
  char item[30];
  char serial[30];
  int quantity;
};
item_inside_struct item_inside;

and want to fill that structure from the data that is coming from an MQTT message with topic "item_inside"

The problem is that I am not sure how to properly parse the data.
I can see that in my callback function, the payload data gets converted into a string:
strncpy (message, (char*)payload, length);

So now my "message" contains all the payload information. How do I efficiently parse it so I can assign my structure to the payload values as following?:

item_inside.item = "item1"
item_inside.serial ="1A2B"
item_inside.quantity = 50

Hi
I think this would be better asked on an Arduino forum as that where those type of experts hang out

I can only suggest the same. In this case @zazas321 need basic support regarding software language C

I'm also using JSON over MQTT for my system, and I use the cJSON library to parse received commands and to create outgoing status and data messages. It's very simple and lightweight, and will easily handle the type of message you've shown.

It's downloadable from GitHub - DaveGamble/cJSON: Ultralightweight JSON parser in ANSI C

edit : a few lines of code to demonstrate its use :

// Parse incoming JSON string
cJSON* jsonObj = cJSON_Parse((char*)(payload));

// Get pointer to specific field in JSON object
cJSON* item = cJSON_GetObjectItem(jsonObj, "quantity");
if (item != NULL)
{
    // Get integer value - other fields allow access to strings, floats etc.
    item_inside.quantity = item->valueint;
}

// Delete JSON object (don't forget this)
cJSON_Delete(jsonObj);
1 Like

Thank you sir! I have downloaded the library and implemented it as following:

if (strcmp(topic,item_inside_topic) == 0){ //if number_to_pick received, means the complecataion has been scanned and initiate the pick_to_light
     cJSON* jsonObj = cJSON_Parse((char*)(payload));
  // Get pointer to specific field in JSON object
    cJSON* item = cJSON_GetObjectItem(jsonObj, "item");
    cJSON* serial = cJSON_GetObjectItem(jsonObj, "serial");
    cJSON* quantity = cJSON_GetObjectItem(jsonObj, "quantity");
    if (item != NULL)
    {
     //Get integer value - other fields allow access to strings, floats etc.
    item_inside.item = item->valuestring;
    item_inside.serial = serial->valuestring;
    item_inside.quantity = quantity->valueint;
    Serial.println(item_inside.item);
    Serial.println(item_inside.serial);
    Serial.println(item_inside.quantity);
    }

// Delete JSON object (don't forget this)
  cJSON_Delete(jsonObj);
  
  }

Maybe you have also tried to create a json object in your program and then publish a MQTT message?

So i have filled my structure with data:
item_inside.item = "item1"
item_inside.serial = "1A2B"
item_inside.quantity = 50

The only variable that will change very often is going to be quantity. Everytime the change in quantity has been detected, I need to send a message back to raspberry PI with the same json, but updated quantity. So lets say in the program, I decrement quantity by 1, then I have to publish a mqtt message:

item_inside_data = {"item":"item_inside.item","serial":"item_inside.serial","quantity":item_inside.quantity}
client.publish(item_inside_topic, item_inside_data); // publish a message when quantity updated 

Your receive code looks OK, and creating a message to send is also fairly easy with cJSON, e.g. :

    // Create a JSON object with the required info
    cJSON* sendObj = cJSON_CreateObject();
    cJSON_AddStringToObject(sendObj, "item", item_inside.item);
    cJSON_AddStringToObject(sendObj, "serial", item_inside.serial);
    cJSON_AddNumberToObject(sendObj, "quantity", item_inside.quantity);

    // Convert to a string
    char *msg = cJSON_Print(sendObj);

    // Send the info
    client.publish(item_inside_topic, (byte*)(msg), strlen(msg));

    // Clean up
    free(msg);
    cJSON_Delete(sendObj);

Note : I use a wrapper round the MQTT library, so your "publish" call may need slightly different parameters.

Thank you again sir! I have implemented that - I am having one problem though. I believe that is something to do with the way I have initialized my structure or my strings item_inside.item and item_inside.serial:

typedef struct {
  char *item;
  char *serial;
  int quantity;
}item_inside_struct;
item_inside_struct item_inside;

As previously mentioned, I am able to fill the structure here without any problems:

  if (strcmp(topic,available_items_topic) == 0){ //if number_to_pick received, means the complecataion has been scanned and initiate the pick_to_light
    available_items = atoi(message);
    //Serial.print("available_items = ");
    //Serial.println(available_items);
    OLED_display(available_items);
    }
    
  if (strcmp(topic,ota_topic) == 0){ 
    start_OTA();    
    }

  if (strcmp(topic,item_inside_topic) == 0){ //if number_to_pick received, means the complecataion has been scanned and initiate the pick_to_light
     cJSON* jsonObj = cJSON_Parse((char*)(payload));
  // Get pointer to specific field in JSON object
    cJSON* item = cJSON_GetObjectItem(jsonObj, "item");
    cJSON* serial = cJSON_GetObjectItem(jsonObj, "serial");
    cJSON* quantity = cJSON_GetObjectItem(jsonObj, "quantity");
    if (item != NULL)
    {
     //Get integer value - other fields allow access to strings, floats etc.
    item_inside.item = item->valuestring;
    item_inside.serial = serial->valuestring;
    item_inside.quantity = quantity->valueint;
    Serial.println(item_inside.item);
    Serial.println(item_inside.serial);
    Serial.println(item_inside.quantity);
    }

When the quantity updates, I am using this function to send the json to the mqtt:

void create_JSON_object()
{
  Serial.println("CREATING JSON OBJECT");
  Serial.println("item_inside.item=");
  Serial.println(item_inside.item);
  Serial.println("item_inside.serial=");
  Serial.println(item_inside.serial);
  Serial.println("item_inside.quantity=");
  Serial.println(item_inside.quantity);

    cJSON* sendObj = cJSON_CreateObject();
    cJSON_AddStringToObject(sendObj, "item", item_inside.item);
    cJSON_AddStringToObject(sendObj, "serial", item_inside.serial);
    cJSON_AddNumberToObject(sendObj, "quantity", item_inside.quantity);

    // Convert to a string
    char *msg = cJSON_Print(sendObj);

    // Send the info
    client.publish(item_inside_topic, (byte*)(msg), strlen(msg));

    // Clean up
    free(msg);
    cJSON_Delete(sendObj);
}

Serial monitor output:

CREATING JSON OBJECT
14:07:43.202 -> item_inside.item=
14:07:43.202 -> ⸮⸮?⸮"
14:07:43.202 -> item_inside.serial=
14:07:43.202 -> ⸮⸮⸮?⸮⸮}⸮-⸮⸮}⸮⸮⸮⸮⸮⸮⸮⸮
14:07:43.202 -> item_inside.quantity=
14:07:43.202 -> 49

As you can see, the quantity is correct, because i decremented by 1. However, the item_inside.item and item_inside.serial data seems corrupted??

How could that happen? I have initialized my structure in global scope and I have checked if i properly assigned the values of my structure here:

    item_inside.item = item->valuestring;
    item_inside.serial = serial->valuestring;
    item_inside.quantity = quantity->valueint;
    Serial.println(item_inside.item);
    Serial.println(item_inside.serial);
    Serial.println(item_inside.quantity);

and it seemed right cause in serial monitor I could see correct values.

I have also noticed slight difference:
When I initially send data from my raspberry PI to my ESP8266 device, it comes like that:

14:09:45.303 -> Message arrived [device1/item_inside] {"item":"item1","serial":"1A2B","quantity":50}

Since my ESP8266 is subscribed to the same topic, i am able to receive the information that the ESP8266 itself sends and it looks like that:

14:09:49.375 -> Message arrived [device1/item_inside] {
14:09:49.375 -> 	"item":	"⸮\u0001",
14:09:49.375 -> 	"serial":	"",
14:09:49.375 -> 	"quantity":	49

Despite the fact that the item and serial data is corrupted, you can notice that it is slightly different format than the original json sent by raspberry PI. Raspberry PI sends a whole message in one line whereas here, every object is in a new line. I am not sure whether that can cause any problems?

Ah, the problem with what appears to be corrupt strings is almost certainly because the "valuestring" field is a pointer to the string stored in the cJson structure, and once it's been deleted is no longer valid.

item_inside.item = item->valuestring;  // copies a pointer to memory which gets deleted

item_inside.quantity = quantity->valueint;  // copies a value

The "valueint" assignment is OK since it's copying a numeric value, but for the strings you need to do a string copy, e.g.

strncpy(item_inside.item, item->valuestring, maxlen);

You'll also need to allocate the struct field as either a fixed (max) size in the declaration, or malloc it to the required size before copying.

Would you be able to tell me what is the purpose of allocating the size of string in the structure declaration? The examples that I have seen usually create a char array ( char* ) type without allocating any size

The char* type is a pointer type, and is 32 bits (4 bytes) on the ESP8266. In use, it points to the address in memory which contains the actual characters of the string.

In a lot of examples strings are created statically, e.g. :

char* str = "This is a string";

which allocates a fixed area in memory for the character string, so copying the pointer value will work :

char* str2 = str;

If you assign a value to it by

item_inside.item = item->valuestring;

then it copies the 4 byte address, but not the contents of the string. When the string gets deleted the memory is available for other use, and may be overwritten (or may be cleared by the delete operation) so the string's contents vanish.

When the string isn't static then you have to check whether it's likely to disappear during processing. This is a quite common cause of errors or crashes :wink: You also need to check the length required when declaring or allocating space for a string, as copying past the end of the allocated memory is another frequent error.

Thank you very much.. This task is getting more and more complicated the more things I try ( most of them dont work :D) . Anyway, thanks.

Also, I do not think I can set a size for item and serial since they both can be variable length. Does that make the solution even more complex?

There are a couple of ways to approach it. Either declare them as much larger than you ever expect or check the size received and reallocate if needed.

Going back to your first post, I see you set them to 30 bytes

struct item_inside_struct {
char item[30];
char serial[30];
int quantity;
};

so if that is larger than you expect to receive, then it should be OK (but I'd recommend either checking before copying or using e.g. "strncpy").

It's also possible to declare them as char* types and allocate an initial size, then use "realloc" to increase the size if the received string is larger, but that would require additional checks and keeping note of the allocated size etc. This is a more memory-efficient method, but more complex to code.

I have replaced the initial structure declaration where I had char item[30] and char serial[30] to char* item and char* serial since I am was doing some string operations such as:
item_inside.serial = serial->valuestring;
strcpy(item_inside.serial,serial->valuestring);

Both of these are not available when I declare a char serial[30]
the error I got:

incompatible types in assignment of 'char*' to 'char [30]'

Also, could you tell me about how to allocate initial size for char* type as I could not find any relevant information on that

Thanks for this thread and the useful information about programming C on the ESP8266. Since I'm not really understanding what I do, I call my programming style frankensteining: putting dead peaces together from the graveyard (the Internet) and power it up hoping it starts to live. It quite often works. :wink:

I use the async MQTT client https://github.com/marvinroger/async-mqtt-client
and this JSON client https://github.com/bblanchon/ArduinoJson

There are enough examples delivered with it for me to be able to create JSON formatted messages to send via the broker to node-red. And decompose incoming JSON MQTT messages.

Since I have no clue about the pointer stuff, I got hit many times with disappeared strings. So I started to copy all the char* stuff into Strings (uppercase S, Arduino programming environment). This is not efficient, its laughed at by the real programmers, but it works reliably, which is good enough for me.

Also, just a very strange thing I noticed when I try to use the string operation as following:
strcpy(item_inside.item,item->valuestring)

The program freezes instantly and returns me this message

-> Message arrived [device1/item_inside] {"item":"item1","serial":"1A2B","quantity":50}
15:27:32.390 -> 
15:27:32.390 -> --------------- CUT HERE FOR EXCEPTION DECODER ---------------
15:27:32.390 -> 
15:27:32.390 -> Exception (29):
15:27:32.390 -> epc1=0x40212bdb epc2=0x00000000 epc3=0x00000000 excvaddr=0x00000000 depc=0x00000000
15:27:32.390 -> 
15:27:32.390 -> >>>stack>>>
15:27:32.390 -> 
15:27:32.390 -> ctx: cont
15:27:32.390 -> sp: 3ffffd40 end: 3fffffc0 offset: 0190
15:27:32.390 -> 3ffffed0:  3fffff1e 3ffef92f 3ffeeb28 402020f4  
15:27:32.390 -> 3ffffee0:  402075f4 3ffef92f 3ffeeb28 40202d2f  
15:27:32.390 -> 3ffffef0:  7469227b 3a226d65 65746922 2c22316d  
15:27:32.423 -> 3fffff00:  72657322 226c6169 4131223a 2c224232  
15:27:32.423 -> 3fffff10:  61757122 7469746e 353a2279 40007d30  
15:27:32.423 -> 3fffff20:  3ffffef0 3ffef943 3fff0294 3ffefd74  
15:27:32.423 -> 3fffff30:  3fffff60 00000001 00000002 00000045  
15:27:32.423 -> 3fffff40:  00001a52 00000013 3ffee7e0 4020ceb8  
15:27:32.423 -> 3fffff50:  00001a52 00000000 3ffee7e0 40206538  
15:27:32.423 -> 3fffff60:  007a1201 2456420e 3ffee800 40205bec  
15:27:32.457 -> 3fffff70:  00000000 00000000 3ffee7e0 40205c0c  
15:27:32.457 -> 3fffff80:  3fffdad0 00000000 3ffee7e0 3ffeec58  
15:27:32.457 -> 3fffff90:  3fffdad0 00000000 3ffee7e0 40202a5c  
15:27:32.457 -> 3fffffa0:  feefeffe feefeffe 3ffeec18 40209910  
15:27:32.457 -> 3fffffb0:  feefeffe feefeffe 3ffe8524 40100fd9  
15:27:32.457 -> <<<stack<<<
15:27:32.457 -> 
15:27:32.457 -> --------------- CUT HERE FOR EXCEPTION DECODER ---------------
15:27:32.491 -> 
15:27:32.491 ->  ets Jan  8 2013,rst cause:2, boot mode:(3,6)
15:27:32.491 -> 
15:27:32.491 -> load 0x4010f000, len 3664, room 16 
15:27:32.524 -> tail 0
15:27:32.524 -> chksum 0xee
15:27:32.524 -> csum 0xee
15:27:32.524 -> v39c79d9b
15:27:32.524 -> ~ld
15:27:34.586 -> 

Hahaha thats very much like me sir. I will definately look into what you suggested. I think I have mqtt client side sorted and have already implemented libraries for that. I just need to sort the json parsing

If you declare it as "char serial[30]" then the assignment

item_inside.serial = serial->valuestring;

will give an error, since the field "valuestring" is a pointer (char*). This will only work if you declare it as a char*, but then you're back to copying a pointer to a string which will be deleted.

The string copy should be OK, since the declaration has allocated 30 bytes of space for the string. (You should however check that the string being copied is less than 30 characters to prevent possible errors.)

It's hard to tell what's caused the crash but I suspect it may be due to copying more characters than have been allowed for (30). Try adding a check, and printing the length of the string to see if that's possibly what's happening.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.