Hello all,
I need help resolving a problem I'm facing when I pass large amounts of data (>110,000 JSON objects) to a change node in my flow.
I get the error "Invalid JSONata expression: Maximum call stack size exceeded". Within the change node, I take the incoming data and format it using JSONata to meet the data format that is required by HighChart.
For data that is less than 110,000 JSON objects, my flow is able to process it and display a plot out using the zoom enabled HighChart.
Here is the part of my flow that is messing up when data gets too large.
[{"id":"99305dcc.69bf6","type":"inject","z":"8b707301.89767","name":"trigger read file","topic":"on","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":140,"y":260,"wires":[["7aecc62a.d5d908"]]},{"id":"7aecc62a.d5d908","type":"file in","z":"8b707301.89767","name":"open http response file","filename":"C:\\Retrieved_PI_data_via_Node_Red_flow\\Temp_Retrieved_data\\http_query_response_150k_JSON.json","format":"utf8","chunk":false,"sendError":false,"encoding":"none","x":280,"y":220,"wires":[["df4d6279.43c37"]]},{"id":"df4d6279.43c37","type":"json","z":"8b707301.89767","name":"","property":"payload","action":"obj","pretty":false,"x":330,"y":260,"wires":[["f652ba75.5c7378","13b6dd6d.1aa0d3"]]},{"id":"13b6dd6d.1aa0d3","type":"debug","z":"8b707301.89767","name":"file in output","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":490,"y":260,"wires":[]},{"id":"f652ba75.5c7378","type":"function","z":"8b707301.89767","name":"Data Size [array]","func":"var data_points = msg.payload;\n\nvar dp_al = data_points.length;\n\nmsg.al = dp_al;\n\nreturn msg;","outputs":1,"noerr":0,"x":150,"y":300,"wires":[["150f7fe2.c1c4e","b281afa3.ed77f"]]},{"id":"150f7fe2.c1c4e","type":"debug","z":"8b707301.89767","name":"array length","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"al","targetType":"msg","x":150,"y":340,"wires":[]},{"id":"b281afa3.ed77f","type":"switch","z":"8b707301.89767","name":"if data >10k plot data using HighChart","property":"al","propertyType":"msg","rules":[{"t":"lt","v":"10000","vt":"num"},{"t":"gte","v":"10000","vt":"num"}],"checkall":"true","repair":false,"outputs":2,"x":430,"y":300,"wires":[[],["c55e1ccb.a1674"]]},{"id":"c55e1ccb.a1674","type":"switch","z":"8b707301.89767","name":"Route msg based on Value data type Obj or other","property":"payload[10].Value","propertyType":"msg","rules":[{"t":"istype","v":"number","vt":"number"},{"t":"istype","v":"object","vt":"object"}],"checkall":"true","repair":false,"outputs":2,"x":470,"y":360,"wires":[["f32549c0.278878","5104f50d.1b028c"],["80faae92.2d8c1","422ce3ad.aa53bc"]]},{"id":"5104f50d.1b028c","type":"change","z":"8b707301.89767","name":"Format http output payload data for HighChart","rules":[{"t":"set","p":"payload","pt":"msg","to":"[$$.payload.\t [\t $toMillis(Timestamp),\t Value\t ]\t]","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":880,"y":340,"wires":[[]]},{"id":"422ce3ad.aa53bc","type":"change","z":"8b707301.89767","name":"Format http output payload data obj for HighChart","rules":[{"t":"set","p":"payload","pt":"msg","to":"[$$.payload.\t [\t $toMillis(Timestamp),\t Value.Value\t ]\t]","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":890,"y":380,"wires":[[]]},{"id":"f32549c0.278878","type":"debug","z":"8b707301.89767","name":"out val","active":true,"tosidebar":true,"console":false,"tostatus":true,"complete":"payload","targetType":"msg","x":750,"y":280,"wires":[]},{"id":"80faae92.2d8c1","type":"debug","z":"8b707301.89767","name":"out obj ","active":true,"tosidebar":true,"console":false,"tostatus":true,"complete":"payload","targetType":"msg","x":750,"y":420,"wires":[]}]
BTW, the data is retrieved via a Web API query to a remote database server within a company network. And the MAX JSON data that can be retrieved per query is 150k. In the interest of this forum, I have saved a retrieved data (dummy data - not real operation data) that has 150,000 JSON objects for your reference. Please, follow the link below as it is too big to be uploaded here. Please scan for malware or virus if you're in doubt.
The path I referenced for the file in my flow is at
"C:\Retrieved_PI_data_via_Node_Red_flow\Temp_Retrieved_data\http_query_response_150k_JSON.json". So, you'd need to put it there in order for my flow to work on your Node-Red setup.
I'm using Node Red 1.0.4, node.js v12.16.1 running on Windows 10, with 8GB RAM and i7 core. And this is what you'll get when you import my flow into your Node Red.
Really look forward to hearing from the Node Red experts in here.