# Parallel Processing of Messages

**URL:** <https://discourse.nodered.org/t/parallel-processing-of-messages/42852>\
**Category:** General\
**Created:** [19 March 2021 06:48 UTC](https://discourse.nodered.org/t/parallel-processing-of-messages/42852 "2021-03-19T06:48:57Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![sathishkumarc14](https://avatars.discourse-cdn.com/v4/letter/s/e480ec/32.png) [@sathishkumarc14](https://discourse.nodered.org/u/sathishkumarc14)\
**Post date:** [19 March 2021 06:48 UTC](https://discourse.nodered.org/t/parallel-processing-of-messages/42852/1 "2021-03-19T06:48:58Z")

</div>

I am having a flow as below,  
TCP in node connected with 5000 clients. Each client sends data in every 60 seconds. Queue holding 20k+ messages over period of time and it is keep on increasing. Is there any methods/nodes there process/inject these messages in parallel.

By using one more database node and processing messages in parallel (Kind of multi threading).

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/f/e/fecdd02be59173b3bb1f6cb12c807c3900378a34.png)

```auto
[{"id":"90d3fc74.22329","type":"tcp in","z":"3c97710c.9cbbce","name":"","server":"server","host":"","port":"1010","datamode":"stream","datatype":"utf8","newline":"","topic":"","base64":false,"x":140,"y":980,"wires":[["1c033601.c229aa"]]},{"id":"1c033601.c229aa","type":"simple-queue","z":"3c97710c.9cbbce","name":"","firstMessageBypass":true,"bypassInterval":"1","x":275,"y":980,"wires":[["5457d7f5.b5b558"]],"l":false},{"id":"556c0060.b5a9d","type":"function","z":"3c97710c.9cbbce","name":"Parse Data","func":"msg.payload = \"{\\\"Data\\\" : \" + msg.payloadData + \",\\\"IP\\\" : \\\"\" + msg.ip + \"\\\"}\" ;\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":530,"y":980,"wires":[["5d530ccf.efa494"]]},{"id":"5457d7f5.b5b558","type":"change","z":"3c97710c.9cbbce","name":"","rules":[{"t":"set","p":"payloadData","pt":"msg","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":395,"y":980,"wires":[["556c0060.b5a9d"]],"l":false},{"id":"5d530ccf.efa494","type":"MSSQL","z":"3c97710c.9cbbce","mssqlCN":"ff8fbd25.aebea","name":"","outField":"payload","returnType":0,"throwErrors":"1","query":"[CPA].[dbo].[InsertCPA]","modeOpt":"","modeOptType":"execute","queryOpt":"","queryOptType":"editor","paramsOpt":"","paramsOptType":"editor","params":[{"output":false,"name":"Data","type":"VarChar(max)","valueType":"msg","value":"payload"}],"x":740,"y":980,"wires":[["75cfacd7.66b864"]]},{"id":"75cfacd7.66b864","type":"debug","z":"3c97710c.9cbbce","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":930,"y":980,"wires":[]},{"id":"ff8fbd25.aebea","type":"MSSQL-CN","tdsVersion":"7_4","name":"local","server":"localhost","port":"1433","encyption":false,"trustServerCertificate":false,"database":"","useUTC":false,"connectTimeout":"15000","requestTimeout":"15000","cancelTimeout":"5000","pool":"10","parseJSON":false,"enableArithAbort":true}]

```

---

<div class="post-metadata">

**Author:** ![hardillb](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/hardillb/32/12373_2.png) [@hardillb](https://discourse.nodered.org/u/hardillb)\
**Post date:** [19 March 2021 07:37 UTC](https://discourse.nodered.org/t/parallel-processing-of-messages/42852/2 "2021-03-19T07:37:25Z")

</div>

The NodeJS runtime that Node-RED runs on top of is inherently single threaded. It can run tasks in parallel but it will only actually work on one at a time and will give up control when ever a task reaches a blocking io call.

What is the queue node actually achieving here? it appears to just be acting as a delay node to rate limit the flow down the flow, if that rate is lower than the input rate it will always lead to a memory leak with backed up messages. If you remove it then the system will process messages as quickly as the database node can handle the inserts.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)\
**Post date:** [18 May 2021 07:38 UTC](https://discourse.nodered.org/t/parallel-processing-of-messages/42852/3 "2021-05-18T07:38:26Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.