Load test with hundreds of JSON messages

General
1

I've only just installed Node RED and I'm looking for a bit of direction for my project
I've created an application for my companies tracking devices that send cell tower and GPS data over mqtt and I was looking to load test it with hundreds of JSON messages a second!!, is this even feasible with Node RED??

thanks in advance

2

Hi @Hoboclown, welcome to the forum

load testing is certainly possible and "hundreds" of JSON messages should be achievable (though it will depend on how you have written flows and how large the data is etc) but I see no reason it cannot handle this volume.

What have you tried?

1 Like
3

If you just want to send the same message repeatedly at that rate then you can do that with an inject node with an appropriate repeat interval set.

4

Thanks for the quick reply.

I've only tried a single message so far from a flow I found in the cookbook.

So, the real world scenario is, we have nearly 1000 "tags" that dial in every 5 - 15 seconds with all the cell towers they can "see" from 1-15 along with a GPS coordinate, if available, and any public WiFi mac addresses, which we send off to a 3rd party service.

I have an application that will sit in between this to intercept, process, forward to 3rd party, intercept the response and finally forward back to the originating tag

So I would need to simulate a large number of devices each with there own unique ID, topic and a randomly generated JSON payload which I can publish to our broker, my app subscribes to the general topic, processes the data etc etc

Right now I just want to see what kind of load my application can handle so I just need to be able to generate a large number of random payloads to send at random intervals

5

Start by building a flow that creates a single message of the type that you need, with appropriate variable data.

6

Thanks Colin,

I have a working flow that starts with an Inject button press, which generates a JSON object using a javascrip function and sends out the message to my MQTT broker with an mqqt out node and listens for the responce with an mqtt in node.

I can just copy paste the javascript function and mqtt nodes to run off the one inject node but that seems rather over kill especially if i need several hundred!!

7

Indeed. You can send more than one msg from a function. Use node.send or return an array.

I would likely configure the inject so that payload is a number (the number of messages to send). In the function, I would likely generate an array of size payload then fill it with random data. After the function node, I would add a delay, set to rate limit mode so as not to crucify the node runtime.

If you share your function (select function, press CTRL-E) I will show you how

8

Like this:

chrome_ShkQqvU2nB

Demo Flow (use CTRL-I to import)

[{"id":"1a5423c1ce9beffa","type":"inject","z":"1791a7bd576b3a15","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"5","payloadType":"num","x":1150,"y":140,"wires":[["528b9a6f617619a2"]]},{"id":"528b9a6f617619a2","type":"function","z":"1791a7bd576b3a15","name":"Generate n Objects","func":"const length = msg.payload\nconst result = []\n\nconst names = [\n    \"Liam\", \"Emma\", \"Noah\", \"Olivia\", \"William\", \"Ava\", \"James\", \"Isabella\", \"Oliver\", \"Sophia\",\n    \"Benjamin\", \"Charlotte\", \"Elijah\", \"Amelia\", \"Lucas\", \"Mia\", \"Mason\", \"Harper\", \"Logan\", \"Evelyn\",\n    \"Alexander\", \"Abigail\", \"Ethan\", \"Emily\", \"Jacob\", \"Ella\", \"Michael\", \"Elizabeth\", \"Daniel\", \"Camila\"\n]\n\n\nfor (let i = 0; i < length; i++) {\n    const id = Math.floor(Math.random() * 100000) + 100000\n    const age = Math.floor(Math.random() * 50) + 20\n    const randomIndex = Math.floor(Math.random() * names.length)\n    const name = names[randomIndex]\n    result.push({ payload: { id, age, name }})\n}\n\nreturn [result]","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":1330,"y":140,"wires":[["0891152534b4b8f4"]]},{"id":"0891152534b4b8f4","type":"delay","z":"1791a7bd576b3a15","name":"","pauseType":"rate","timeout":"5","timeoutUnits":"seconds","rate":"3","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"allowrate":false,"outputs":1,"x":1190,"y":220,"wires":[["4486980fafe9ce8c"]]},{"id":"4486980fafe9ce8c","type":"debug","z":"1791a7bd576b3a15","name":"debug 25","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1360,"y":220,"wires":[]}]
9

MQTT_Simulator.json (6.6 KB)

this is what i have so far

10

I see where your thinking is going though.

many thnaks

11

for small flows, it is better to paste the JSON export into a code block:

```
type or paste code here
```

NOTE also, in your function, you are stringifying the JS Object before sending it to MQTT node. There is no need to do that (the MQTT node takes care of it)

12

Good catch, i was doing that because the recieving application takes the sequenceNUmber in as number and the MQTT out doesnt like the BigInt. But i've updated the reciving app to take in a string and set the nodes back to send the JSON obeject with the sequenceNumber as a string now.

I appreciate all the help you given.

So, if i wanted to ramp this up to say, 100 devices would you suggest using say, 10 funtions spitting out 10 devices each?? or could i use one function, with multiple outputs each it its own mqqt out node??

13

depends if the message structure is the same or different. If the same, i would write one function and create a visual subroutine using link-call - to minimise copy-pasta code!

e.g...
chrome_SgUVwZSFbj

[{"id":"1a5423c1ce9beffa","type":"inject","z":"f31d05a2a56e6fd6","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"device1","payload":"5","payloadType":"num","x":260,"y":1200,"wires":[["0def5e23a4e21d38"]]},{"id":"0891152534b4b8f4","type":"delay","z":"f31d05a2a56e6fd6","name":"","pauseType":"rate","timeout":"5","timeoutUnits":"seconds","rate":"3","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"allowrate":false,"outputs":1,"x":840,"y":1200,"wires":[["4486980fafe9ce8c"]]},{"id":"4486980fafe9ce8c","type":"debug","z":"f31d05a2a56e6fd6","name":"debug 25","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1010,"y":1200,"wires":[]},{"id":"528b9a6f617619a2","type":"function","z":"f31d05a2a56e6fd6","name":"Generate n Objects","func":"const length = msg.payload\nconst result = []\n\nconst names = [\n    \"Liam\", \"Emma\", \"Noah\", \"Olivia\", \"William\", \"Ava\", \"James\", \"Isabella\", \"Oliver\", \"Sophia\",\n    \"Benjamin\", \"Charlotte\", \"Elijah\", \"Amelia\", \"Lucas\", \"Mia\", \"Mason\", \"Harper\", \"Logan\", \"Evelyn\",\n    \"Alexander\", \"Abigail\", \"Ethan\", \"Emily\", \"Jacob\", \"Ella\", \"Michael\", \"Elizabeth\", \"Daniel\", \"Camila\"\n]\n\n\nfor (let i = 0; i < length; i++) {\n    const id = Math.floor(Math.random() * 100000) + 100000\n    const age = Math.floor(Math.random() * 50) + 20\n    const randomIndex = Math.floor(Math.random() * names.length)\n    const name = names[randomIndex]\n    result.push({ device: msg.topic, id, age, name })\n}\nmsg.payload = result\nreturn msg","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":590,"y":1380,"wires":[["392323d1953dd6f7"]]},{"id":"517c630507adb5f1","type":"link in","z":"f31d05a2a56e6fd6","name":"generate-n-objects","links":[],"x":465,"y":1380,"wires":[["528b9a6f617619a2"]]},{"id":"392323d1953dd6f7","type":"link out","z":"f31d05a2a56e6fd6","name":"link out 2","mode":"return","links":[],"x":725,"y":1380,"wires":[]},{"id":"0def5e23a4e21d38","type":"link call","z":"f31d05a2a56e6fd6","name":"","links":["517c630507adb5f1"],"linkType":"static","timeout":"30","x":440,"y":1200,"wires":[["6ad0290b84e8cba7"]]},{"id":"6ad0290b84e8cba7","type":"split","z":"f31d05a2a56e6fd6","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","property":"payload","x":640,"y":1200,"wires":[["0891152534b4b8f4"]]},{"id":"f99d0e84ed77a345","type":"inject","z":"f31d05a2a56e6fd6","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"device2","payload":"7","payloadType":"num","x":260,"y":1260,"wires":[["8fad238f9cacc632"]]},{"id":"102a7ee57044c881","type":"delay","z":"f31d05a2a56e6fd6","name":"","pauseType":"rate","timeout":"5","timeoutUnits":"seconds","rate":"3","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"allowrate":false,"outputs":1,"x":840,"y":1260,"wires":[["6d037d309cedbac8"]]},{"id":"6d037d309cedbac8","type":"debug","z":"f31d05a2a56e6fd6","name":"debug 4","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1010,"y":1260,"wires":[]},{"id":"8fad238f9cacc632","type":"link call","z":"f31d05a2a56e6fd6","name":"","links":["517c630507adb5f1"],"linkType":"static","timeout":"30","x":440,"y":1260,"wires":[["429881f8382ae51c"]]},{"id":"429881f8382ae51c","type":"split","z":"f31d05a2a56e6fd6","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","property":"payload","x":640,"y":1260,"wires":[["102a7ee57044c881"]]}]
14

Is the listening application also running on the same NR server ?

If so then this wouldn't really be a fair test, as under normal operation it would only be listening and not also generating the messages ?

15

no the reciveing application sits on a server, and its job is to take the JSON from the MQTT payload, process it and send back a response. The messages will ultimatly be coming from handheld tracking devices. These tests are just to product a high number of MQTT messages to see how it fairs before i relase it into the wild. Theoretically we could have 1000's of messages per second comming in and going out an we need to be able to scale the processing

the message structure will largely stay the same, thenumber of cell towers and/or wifi hotspots will increase, but these are in an array. I like that Link-call node, thats way better than having multiple functions i'd have to update.

from what i understand, the generate n object function outputs one long message, and the split literily splits them into seperate ones?? in your previous example Steve you didnt have a split

From your example i see 2 flows one for each device, but reusing the same function, again i could copy past them to have 10/20/50/100 devices but i sense there is way more refined way?

my MQTT topic string contains the unique deviceID, so i could have a linked function to generate these topics have another linked function to the generate the JSON payload and combine them, pass in the nodeRED topic & payload into the MQTT out node and bobs your uncle??

16

With this number of towers, topics and data, I'd for sure be looking very very hard at using SparkplugB vs MQTT.
The pro buffer compression, unified namespace and statefullness will be well worth getting it running at the start vs wishing you'd gone that way at tower 148.

Source. I use MQTT to send / receive ACARS messages from 2000+ aircraft at any time via Node-RED. SPb would have saved me sooooo much time, data and grief, but too late now.

17

Just Looking at SparkPlugB and from what i can tell its a standardization layer on top of MQTT, amongst other things. I'll deffo take a look to see if its suitable for our project, thanks for the heads up