Hi. Noob here.
I came across a task that was not very suitable to node-red, the first and only for me. I used node-red to search for strings among more than 50k plus items in a array, the file is up to 16MB(image from flow below).
The problem is that is taking over 10 seconds to do that, and sometimes the page that is supposed to display the information “times out”, it time out when more than 3 or 4 requests comes at the same time since the last request will take at least 40 seconds and the page timeout at 30 seconds.
My question is… Is there a way to delegate some of this request to other thread(core, processor, worker...) so more than one request can be dealt at the same time?
I´m using a simple for loop inside the function node. Here a simplified image from that.
Cant you put an index on the mongodb and use SQL like queries instead of looping large data
This will massively speed it up.
Also, unless you post your flow, we cant see if there are any major quick wins in your code.
1 Like
I believe the "slowness" is because node(and node-red) are single threaded.
I believe that someone is going to suggest that I delegate the logic to a python code, but I don´t know any other language besides javaScript(yet).
I´m running node-red in a electron, so I was wondering🤔 if is there a way to delegate to a worker in the editor. I run in electron so the editor is aways open.
That is a nice idea. I´ll try to learn how to do it.
I know zero about Databases, probably will take a long time to learn, but will be worth it.
Any suggestion to where to start?
here and here is where I would start.
The key point is a database is often FAR more efficient at finding things than a custom loop (since they use indexes and algorithms out of my pay bracket to achieve this - its their bread and butter).
If you use the databases capabilities this point is not necessarily true as the DB engine will likely be a compiled separate work horse)
Install node-red as a service so you can close the editor.
Did as you told me, I was surprised that it was not that faster.... But, a big but, node-red was able to keep receiving requests normally and the result was no timeouts at all.
Thank you very much it was very helpful.
Did you apply index to the field you are searching? And do the search in the DB ?
Yes. Now is closer to 5 seconds, but it is fine since node-red can keep working on other tasks while it waits for the DB.
1 Like
5s rather than >10s is over 50% faster! That seems like quite a bit to me.
Thing is, you are searching through 50,000 items for "strings". On a $30 computer! 
If you want faster, you need to find out what is slowing things down. Quite probable that running MongoDB along site Node-RED and whatever else you have running is going to cause some tension between the applications. Have a look to see if there is any other software running that you can get rid of. If you are running the Pi's desktop, that is the best place to start. Ditch it and you may find things speed up a fair bit. However, you might also want to look at the docs for MongoDB and look at its minimum requirements. You may well find that the db is having to hit the physical storage as it searches which is especially slow on a Pi due to the use of the SD-Card. Replacing the card storage with an SSD will massively improve throughput in that case though giving MongoDB more RAM will do even more.
1 Like
If you are doing something this big - why not move it from the Electron to an Intel based system running Node-red - lots of small thin options out there that have low power requirements and that will churn through the sort of dataset you are talking about - more importantly all of them will have options for integrating SSDs or faster hard drives through suitable interfaces
Craig
1 Like
I moved everything to Heroku services and mlab Mongo DB, it's fasssst.
Where is the 5s response time though? Is it from mongo, is it the array loop or is it the generating of mass amounts html strings before the request responds? I'd suspect its template generation, you effectively have 2 very large loops, one of which is generating a massive string of text for each result.
Also, why generate the html on nodered side? if you are making a html request, its a better bet to generate the html from a JSON response on the client side?
10s was from array loop, 5s from local machine, now is below 5s on mlab.
All the data is in the database, the user give just a small string as a input and receive back a small web page with a list of itens.
