Parallel Processing of Messages

General
1

I am having a flow as below,
TCP in node connected with 5000 clients. Each client sends data in every 60 seconds. Queue holding 20k+ messages over period of time and it is keep on increasing. Is there any methods/nodes there process/inject these messages in parallel.

By using one more database node and processing messages in parallel (Kind of multi threading).

image
image947×116 7.74 KB 

[{"id":"90d3fc74.22329","type":"tcp in","z":"3c97710c.9cbbce","name":"","server":"server","host":"","port":"1010","datamode":"stream","datatype":"utf8","newline":"","topic":"","base64":false,"x":140,"y":980,"wires":[["1c033601.c229aa"]]},{"id":"1c033601.c229aa","type":"simple-queue","z":"3c97710c.9cbbce","name":"","firstMessageBypass":true,"bypassInterval":"1","x":275,"y":980,"wires":[["5457d7f5.b5b558"]],"l":false},{"id":"556c0060.b5a9d","type":"function","z":"3c97710c.9cbbce","name":"Parse Data","func":"msg.payload = \"{\\\"Data\\\" : \" + msg.payloadData + \",\\\"IP\\\" : \\\"\" + msg.ip + \"\\\"}\" ;\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":530,"y":980,"wires":[["5d530ccf.efa494"]]},{"id":"5457d7f5.b5b558","type":"change","z":"3c97710c.9cbbce","name":"","rules":[{"t":"set","p":"payloadData","pt":"msg","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":395,"y":980,"wires":[["556c0060.b5a9d"]],"l":false},{"id":"5d530ccf.efa494","type":"MSSQL","z":"3c97710c.9cbbce","mssqlCN":"ff8fbd25.aebea","name":"","outField":"payload","returnType":0,"throwErrors":"1","query":"[CPA].[dbo].[InsertCPA]","modeOpt":"","modeOptType":"execute","queryOpt":"","queryOptType":"editor","paramsOpt":"","paramsOptType":"editor","params":[{"output":false,"name":"Data","type":"VarChar(max)","valueType":"msg","value":"payload"}],"x":740,"y":980,"wires":[["75cfacd7.66b864"]]},{"id":"75cfacd7.66b864","type":"debug","z":"3c97710c.9cbbce","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":930,"y":980,"wires":[]},{"id":"ff8fbd25.aebea","type":"MSSQL-CN","tdsVersion":"7_4","name":"local","server":"localhost","port":"1433","encyption":false,"trustServerCertificate":false,"database":"","useUTC":false,"connectTimeout":"15000","requestTimeout":"15000","cancelTimeout":"5000","pool":"10","parseJSON":false,"enableArithAbort":true}]
2

The NodeJS runtime that Node-RED runs on top of is inherently single threaded. It can run tasks in parallel but it will only actually work on one at a time and will give up control when ever a task reaches a blocking io call.

What is the queue node actually achieving here? it appears to just be acting as a delay node to rate limit the flow down the flow, if that rate is lower than the input rate it will always lead to a memory leak with backed up messages. If you remove it then the system will process messages as quickly as the database node can handle the inserts.

3

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.