Good evening everyone. I'm working on a flow to create a bot to retrieve a lot of datas from a website: it loops a http request via the nbrowser node, and each time it retrieves the data by doing a few steps. For reason I don't understand, sometimes the call freezes at some point: sometimes the cli…

Though maybe off topic a bit, but to me it is begging to be asked: Why that many and why so quickly? 3,000 to 4,000 is a lot of calls. I only put the , in there to help quantify the size. I thought (at first glance) it was 30,000 which is a whole other ball game.

I have a list of VAT numbers (i think that's the correct translation? English is not my first language), and I need to get the related addresses of the corresponding companies. What do you mean with "why so quickly?"

Well, if you were doing the 3000 lookups over a longer time it may not be a problem. (I am not really understanding the entire problem. I'm only asking because of what I see) My thoughts are like this: You have to do what ever it is you do to all these addresses. It makes sense to automate it. …

It does not make all the calls at once: I put all the VAT numbers in an array, and then cycle it, making one call at a time for 3000 times. I thought about simply re-injecting the flow when it crashes, but it does it after 20-30 calls so it's not good enough: I need the bot to be usable once a month…

[image] mrark: So we get to the question: can I remove the maximum number of catches allowed by the catch node? Thanks for any help or suggestion! No - you cannot alter this, it is hard coded at 10, and is to stop Node RED from igniting and causing further instability, You have an error loop…

Oh, ok. I see there is another reply but just quickly.... So with what you say, there may be a problem with how the program releases memory after each call. I know you are using Node-red, but....... Each time you call the function to process the data from each customer, memory is used. Then whe…

And just to plug one of my nodes to facilitate this type of control. :angel: it provides the same thing that @Colin wrote - just in a node collection with extras, such as a built in fail-safe [image] @marcus-j-davies/node-red-semaphore-plus An advanced set of semaphore Nodes for Node R…

The flow has a fail safe also. But yes.

I am trying to use this solution, but I don't really get how it works (I'm kinda new to node-red, so sorry for my ignorance ^^). Could you give me a quick tutorial? I don't get how this could fix my problem: if the flow gets an error, wouldn't it stop anyway?

Remove "maximum number of catches"

General

dceejay 25 November 2023 21:57 8

You may want to use a queue to only fire off the next request once one has completed successfully to prevent a DDOS on the serving end... maybe something like this Node red flow queuing - #3 by Colin - that flow is self throttling and goes as fast as it can - but one at a time.

Topic		Replies	Views
How to catch 1 error many times General	5	985	12 February 2021
Debug node causing RangeError: Maximum call stack size exceeded General	28	3272	6 June 2020
Node-Red crashes with RangeError: Maximum call stack size exceeded General	25	4506	15 March 2020
Catch node not catching General	17	2808	1 October 2018
[puppeteer-page-waitFor] RangeError: Maximum call stack size exceeded General	0	60	19 February 2025

Remove "maximum number of catches"

Related topics