Http request node => Invalid message body, was specified to be CBOR but could not decode message (Failed to parse)

Core Development
1

I am trying to use the http request node to post a message with a CBOR binary payload to http://ingestion.edgeimpulse.com/api/training/data

The problem is that I am getting the following error response back:

Invalid message body, was specified to be CBOR but could not decode message (Failed to parse)

FYI this is input message for the http request node:

{
 "_msgid": "c7b700d1.8c935",
 "topic": "",
 "payload": [
   163,
   105,
   112,
   114,
   111,
   116,
   101,
   99,
   116,
   101,
   100,
   163,
   99,
   118,
   ...
 ],
 "headers": {
   "x-api-key": "ei_f2644XXXXXXf",
   "x-file-name": "idle.01",
   "x-label": "idle",
   "Content-Type": "application/cbor"
 }
}

The payload is a buffer of binary data but when I post the same payload to a http in node and get it dumbed by a debug node then it seems to be converted to a string:

image

FYI this is the configuration of the http request node:

image
image631Ă—564 22.1 KB

Note also that I checked the first part of the payload using http://cbor.me/ and this is indeed having the proper CBOR format.

So I am currently thinking that the binary payload gets converted to a string before it gets processed by the destination application.

2

I am just wondering if the http request supports the posting of a binary body.

When very quickly checking the code : https://github.com/node-red/node-red/blob/master/packages/node_modules/%40node-red/nodes/core/network/21-httprequest.js

I see the following:

image
image911Ă—763 97.6 KB

Doesn't this say that it is stringifying my binary payload before sending ?

3

Spot on, turning your payload into a JSON string specifically. I’m not familiar with CBOR, but can you change it into a Buffer? Line 244 would hit in that case

1 Like
4

Thanks, good response. I overlooked that line.
it is indeed a buffer so it is not stringifying my payload.
So the issue must be somewhere else (maybe outside node-red).

5

I noticed that there is a difference when using Content-Type audio/wav instead of application/cbor.

When using audio/wav the buffer send is also received as a buffer by the http in node.
When using application/cbor the buffer send is received as a string in the http in node.

Here below the test flow demonstrating the difference:

[{"id":"c8a363d6.9d7ea","type":"tab","label":"Flow 2","disabled":false,"info":""},{"id":"57f23b8e.96c0d4","type":"inject","z":"c8a363d6.9d7ea","name":"test2","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":110,"y":140,"wires":[["472840dc.7cc27"]]},{"id":"f213a3fd.a5656","type":"http request","z":"c8a363d6.9d7ea","name":"","method":"POST","ret":"txt","paytoqs":false,"url":"http://localhost:1880/test1","tls":"","persist":false,"proxy":"","authType":"","x":310,"y":300,"wires":[["d3095d49.e1207"]]},{"id":"c0f4f4ca.6987e8","type":"http in","z":"c8a363d6.9d7ea","name":"","url":"/test1","method":"post","upload":false,"swaggerDoc":"","x":290,"y":460,"wires":[["b91f8f18.38633","538a872.9174b78","73a5dd8e.eeb734"]]},{"id":"b91f8f18.38633","type":"http response","z":"c8a363d6.9d7ea","name":"","statusCode":"","headers":{},"x":560,"y":460,"wires":[]},{"id":"d3095d49.e1207","type":"debug","z":"c8a363d6.9d7ea","name":"http request response","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":600,"y":300,"wires":[]},{"id":"538a872.9174b78","type":"debug","z":"c8a363d6.9d7ea","name":"received message","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":460,"y":420,"wires":[]},{"id":"bed7e6e9.97b918","type":"function","z":"c8a363d6.9d7ea","name":"set payload to buffer","func":"msg.payload = Buffer.from('00ff', 'hex');\nreturn msg;","outputs":1,"noerr":0,"x":340,"y":220,"wires":[["8ea5db8c.5d44c8","243e5dfa.00aa12","f213a3fd.a5656"]]},{"id":"243e5dfa.00aa12","type":"debug","z":"c8a363d6.9d7ea","name":"http request in","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":580,"y":260,"wires":[]},{"id":"73a5dd8e.eeb734","type":"function","z":"c8a363d6.9d7ea","name":"","func":"msg.payload = msg.payload.toString('hex');\nreturn msg;","outputs":1,"noerr":0,"x":430,"y":520,"wires":[["350dd73e.bc2168"]]},{"id":"350dd73e.bc2168","type":"debug","z":"c8a363d6.9d7ea","name":"payload (received) in hex format","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":660,"y":520,"wires":[]},{"id":"8ea5db8c.5d44c8","type":"function","z":"c8a363d6.9d7ea","name":"","func":"msg.payload = msg.payload.toString('hex');\nreturn msg;","outputs":1,"noerr":0,"x":570,"y":220,"wires":[["ba52257d.b84c58"]]},{"id":"ba52257d.b84c58","type":"debug","z":"c8a363d6.9d7ea","name":"payload (being send) in hex format","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":810,"y":220,"wires":[]},{"id":"7dda9258.f6379c","type":"change","z":"c8a363d6.9d7ea","name":"set msg.headers  - content-type audio/wav","rules":[{"t":"set","p":"headers","pt":"msg","to":"{\"Content-Type\":\"audio/wav\",\"x-label\":\"idle\"}","tot":"json"}],"action":"","property":"","from":"","to":"","reg":false,"x":410,"y":100,"wires":[["bed7e6e9.97b918"]]},{"id":"33da16dd.9c0e9a","type":"inject","z":"c8a363d6.9d7ea","name":"test1","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":110,"y":100,"wires":[["7dda9258.f6379c"]]},{"id":"472840dc.7cc27","type":"change","z":"c8a363d6.9d7ea","name":"set msg.headers  - content-type application/cbor","rules":[{"t":"set","p":"headers","pt":"msg","to":"{\"Content-Type\":\"application/cbor\",\"x-label\":\"idle\"}","tot":"json"}],"action":"","property":"","from":"","to":"","reg":false,"x":420,"y":140,"wires":[["bed7e6e9.97b918"]]}]

So clicking on test1 gives the expected result:

image
image1466Ă—571 97.1 KB

Clicking on test2 doesn't give the expected result:

image
image1470Ă—556 95.5 KB

6

Are you certain you’re already inserting a buffer? Looking at your object, your payload is a regular array, and the http-request is set to respond with a buffer.

7

The payload send is a buffer (see screenshot below) and this can also be seen in the debug tabs in my previous comment for test1 and test2.

image
image1904Ă—556 74.7 KB

Setting http-request return to a binary buffer or a UTF-8 string doesn't seem to make a difference in terms of the message being received by the http in node.

1 Like
8

I think I have found the root cause.
It is the http in node that is converting the body to a string in case content-type is application/cbor

image
image756Ă—720 76.3 KB

9

If I change the content-type from application/cbor into application/octet-stream then the test program is working as expected (ie. the buffer send is also received as a buffer).

Moreover this change also fixed the issue when sending data to edge impulse (http://ingestion.edgeimpulse.com/api/training/data).

10

@janvda Two thoughts. If application/cbor is valid type (which it seems to be) then should we handle it properly as binary ? And as cbor seems to be proper standard would a cbor convertor node be useful ? (Eg like the msgpack one or base64 etc ?)

1 Like
11

Yes, CBOR format is a binary format so it should not be stringified.

I have only identified a problem with the code of the http in core node that is stringifying the buffer
... but for me it was also not working when sending it to the edge impulse ingestion site. In that scenario I am not using http in node, I am only using http request node.
So besides the issue with the http in there is also another issue (maybe not in node-red but on the edge impulse ingestion site).

Anyway, I managed to fix it by using content-type application/octet-stream. So for me this solution is fine.

I am using node-red-contrib-cbor for the conversion from javascript object to cbor and that is working fine.

... but there was one tricky thing. The cbor format used by edge impulse makes use of an indefinite length array which doesn't have a javascript counterpart. So in order to create the proper indefinite array, I had first create a standard array and then updated some bytes in the buffer to transform it into an indefinite length array.
Here the code:

/* assured that it should end like:
     9F                                # array(*)
         6E                             # text(14)
            5265662D42494E4152592D693136 # "Ref-BINARY-i16"
         FF                             # primitive(*)
*/

var buf = msg.payload

buf.writeUInt8(0x9f, buf.length - 17);
buf.writeUInt8(0x6e, buf.length - 16);
buf.writeUInt8(0xff, buf.length - 1);

return msg;
12

Yes ok - so yes we should fix http in, but can't do anything about other servers of course... and great - I didn't spot that contrib node already exists.

1 Like