# Remove "maximum number of catches"

**URL:** <https://discourse.nodered.org/t/remove-maximum-number-of-catches/83187>\
**Category:** General\
**Created:** [25 November 2023 21:07 UTC](https://discourse.nodered.org/t/remove-maximum-number-of-catches/83187 "2023-11-25T21:07:16Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![mrark](https://avatars.discourse-cdn.com/v4/letter/m/b9bd4f/32.png) [@mrark](https://discourse.nodered.org/u/mrark)\
**Post date:** [25 November 2023 21:07 UTC](https://discourse.nodered.org/t/remove-maximum-number-of-catches/83187/1 "2023-11-25T21:07:16Z")

</div>

Good evening everyone.

I'm working on a flow to create a bot to retrieve a lot of datas from a website: it loops a http request via the nbrowser node, and each time it retrieves the data by doing a few steps. For reason I don't understand, sometimes the call freezes at some point: sometimes the click doesn't bring me to the right page but just refreshes the current one, and sometimes the bot is unable to retrieve the right selector when in the correct page. I have no idea why.  
That wouldn't be a big deal, because I built it trying to go around this problem, inserting a catch: so, whenever an error occurs, whatever error it might be, the node simply restarts.

And that's good.  
The problem is that I have to do this operation thousands of times (about 3000/4000 addresses), so after too few calls, the flow stops working, giving me this error:

"Message exceeded maximum number of catches"

So we get to the question: can I remove the maximum number of catches allowed by the catch node?  
Thanks for any help or suggestion!

---

<div class="post-metadata">

**Author:** ![Trying\_to\_learn](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/trying_to_learn/32/28400_2.png) [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)\
**Post date:** [25 November 2023 21:24 UTC](https://discourse.nodered.org/t/remove-maximum-number-of-catches/83187/2 "2023-11-25T21:24:59Z")

</div>

Though maybe off topic a bit, but to me it is begging to be asked:

Why that many and why so quickly?  
`3,000` to `4,000` is a lot of calls.  
I only put the `,` in there to help quantify the size.

_I thought (at first glance) it was `30,000` which is a whole other ball game._

---

<div class="post-metadata">

**Author:** ![mrark](https://avatars.discourse-cdn.com/v4/letter/m/b9bd4f/32.png) [@mrark](https://discourse.nodered.org/u/mrark)\
**Post date:** [25 November 2023 21:33 UTC](https://discourse.nodered.org/t/remove-maximum-number-of-catches/83187/3 "2023-11-25T21:33:32Z")

</div>

I have a list of VAT numbers (i think that's the correct translation? English is not my first language), and I need to get the related addresses of the corresponding companies.  
What do you mean with "why so quickly?"

---

<div class="post-metadata">

**Author:** ![Trying\_to\_learn](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/trying_to_learn/32/28400_2.png) [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)\
**Post date:** [25 November 2023 21:39 UTC](https://discourse.nodered.org/t/remove-maximum-number-of-catches/83187/4 "2023-11-25T21:39:13Z")

</div>

Well, if you were doing the _3000_ lookups over a longer time it may not be a problem.

(I am not really understanding the entire problem. I'm only asking because of what I see)

My thoughts are like this:  
You have to do _what ever it is you do_ to all these addresses.  
It makes sense to automate it.

But if you get the computer to do them ALL at once - with that many to do - it could run out of memory.

Can you brake it down into two smaller lists and do it in two passes?  
(or maybe 3)

That way you may not run out of space and _blow_ the catch number.

---

<div class="post-metadata">

**Author:** ![mrark](https://avatars.discourse-cdn.com/v4/letter/m/b9bd4f/32.png) [@mrark](https://discourse.nodered.org/u/mrark)\
**Post date:** [25 November 2023 21:43 UTC](https://discourse.nodered.org/t/remove-maximum-number-of-catches/83187/5 "2023-11-25T21:43:40Z")

</div>

It does not make all the calls at once: I put all the VAT numbers in an array, and then cycle it, making one call at a time for 3000 times. I thought about simply re-injecting the flow when it crashes, but it does it after 20-30 calls so it's not good enough: I need the bot to be usable once a month or so, with similar numbers. So this is why I need a way to go around the maximum number of catches.

---

<div class="post-metadata">

**Author:** ![marcus-j-davies](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/marcus-j-davies/32/103435_2.png) [@marcus-j-davies](https://discourse.nodered.org/u/marcus-j-davies)\
**Post date:** [25 November 2023 21:45 UTC](https://discourse.nodered.org/t/remove-maximum-number-of-catches/83187/6 "2023-11-25T21:45:54Z")

</div>

> [@mrark](#):
>
> So we get to the question: can I remove the maximum number of catches allowed by the catch node?  
> Thanks for any help or suggestion!

No - you cannot alter this, it is hard coded at 10, and is to stop Node RED from igniting and causing further instability, You have an error loop that needs to be fixed.

Personally, and I mean this in the nicest way, you should really fix the fault at source, and not try to avoid Node RED error handling.

---

<div class="post-metadata">

**Author:** ![Trying\_to\_learn](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/trying_to_learn/32/28400_2.png) [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)\
**Post date:** [25 November 2023 21:46 UTC](https://discourse.nodered.org/t/remove-maximum-number-of-catches/83187/7 "2023-11-25T21:46:09Z")

</div>

Oh, ok.

I see there is another reply but just quickly....

So with what you say, there may be a problem with how the _program_ releases memory after each call.

I know you are using Node-red, but.......

Each time you _call_ the _function_ to process the data from each customer, memory is used.  
Then when complete the memory should be released.

If it isn't being released then eventually you will run out of memory.

That's the best way I can explain what I suspect COULD be the problem.

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [25 November 2023 21:57 UTC](https://discourse.nodered.org/t/remove-maximum-number-of-catches/83187/8 "2023-11-25T21:57:15Z")

</div>

You may want to use a queue to only fire off the next request once one has completed successfully to prevent a DDOS on the serving end... maybe something like this [Node red flow queuing - #3 by Colin](https://discourse.nodered.org/t/node-red-flow-queuing/78630/3) - that flow is self throttling and goes as fast as it can - but one at a time.

---

<div class="post-metadata">

**Author:** ![marcus-j-davies](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/marcus-j-davies/32/103435_2.png) [@marcus-j-davies](https://discourse.nodered.org/u/marcus-j-davies)\
**Post date:** [25 November 2023 22:01 UTC](https://discourse.nodered.org/t/remove-maximum-number-of-catches/83187/10 "2023-11-25T22:01:57Z")

</div>

And just to plug one of my nodes to facilitate this type of control. 👼

it provides the same thing that @Colin wrote - just in a node collection with extras, such as a built in fail-safe

> **[@marcus-j-davies/node-red-semaphore-plus](https://flows.nodered.org/node/@marcus-j-davies/node-red-semaphore-plus)**
>
> An advanced set of semaphore Nodes for Node RED, with a dynamic fail-safe

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [25 November 2023 22:29 UTC](https://discourse.nodered.org/t/remove-maximum-number-of-catches/83187/11 "2023-11-25T22:29:37Z")

</div>

The flow has a fail safe also. But yes.

---

<div class="post-metadata">

**Author:** ![mrark](https://avatars.discourse-cdn.com/v4/letter/m/b9bd4f/32.png) [@mrark](https://discourse.nodered.org/u/mrark)\
**Post date:** [25 November 2023 23:32 UTC](https://discourse.nodered.org/t/remove-maximum-number-of-catches/83187/12 "2023-11-25T23:32:07Z")

</div>

I am trying to use this solution, but I don't really get how it works (I'm kinda new to node-red, so sorry for my ignorance ^^).

Could you give me a quick tutorial? I don't get how this could fix my problem: if the flow gets an error, wouldn't it stop anyway?

---

<div class="post-metadata">

**Author:** ![mrark](https://avatars.discourse-cdn.com/v4/letter/m/b9bd4f/32.png) [@mrark](https://discourse.nodered.org/u/mrark)\
**Post date:** [25 November 2023 23:33 UTC](https://discourse.nodered.org/t/remove-maximum-number-of-catches/83187/13 "2023-11-25T23:33:26Z")

</div>

I tried fixing the error of course 😭  
I really don't understand why those errors occur. It looks like it depends from the website, because they look completely random.

---

<div class="post-metadata">

**Author:** ![Sean-McG](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/sean-mcg/32/54677_2.png) [@Sean-McG](https://discourse.nodered.org/u/Sean-McG)\
**Post date:** [26 November 2023 03:31 UTC](https://discourse.nodered.org/t/remove-maximum-number-of-catches/83187/14 "2023-11-26T03:31:52Z")

</div>

What have you found out about the error source, as suggested it could be the website blocking you in some way due to unusual traffic ?

---

<div class="post-metadata">

**Author:** ![marcus-j-davies](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/marcus-j-davies/32/103435_2.png) [@marcus-j-davies](https://discourse.nodered.org/u/marcus-j-davies)\
**Post date:** [26 November 2023 08:55 UTC](https://discourse.nodered.org/t/remove-maximum-number-of-catches/83187/15 "2023-11-26T08:55:13Z")

</div>

What service are you using to obtain the address?  
You realise there is a Web Service for this type of resource retrieval?

Im bringing it up, as you seem to be using web scraping.

> [@mrark](#):
>
> sometimes the call freezes at some point: sometimes the click doesn't bring me to the right page but just refreshes the current one

[https://developer.service.hmrc.gov.uk/api-documentation/docs/api/service/vat-registered-companies-api/1.0](https://developer.service.hmrc.gov.uk/api-documentation/docs/api/service/vat-registered-companies-api/1.0)

The return provides the required info.

```auto
{
  "target": {
    "name": "Credite Sberger Donal Inc.",
    "vatNumber": "553557881",
    "address": {
      "line1": "131B Barton Hamlet",
      "postcode": "SW97 5CK",
      "countryCode": "GB"
    }
  },
  "processingDate": "2019-01-31T12:43:17+00:00"
}

```

I realise the is a UK web service, but its worth checking to see if something similar exists with your government, if not residing in the UK

Further more, I would look at building a database of VAT-\> Address's, so overtime you can cut down on the number of requests as your DB gets populated

---

<div class="post-metadata">

**Author:** ![mrark](https://avatars.discourse-cdn.com/v4/letter/m/b9bd4f/32.png) [@mrark](https://discourse.nodered.org/u/mrark)\
**Post date:** [27 November 2023 02:12 UTC](https://discourse.nodered.org/t/remove-maximum-number-of-catches/83187/16 "2023-11-27T02:12:27Z")

</div>

> [@marcus-j-davies](#):
>
> inging i

I found lots of API sources doing what you say, but they are not free. I was looking for a way to do it with node red, but it seems like it is not the best tool to do such a big web scraping...

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [27 November 2023 10:01 UTC](https://discourse.nodered.org/t/remove-maximum-number-of-catches/83187/17 "2023-11-27T10:01:14Z")

</div>

Which node is generating the errors you are catching?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)\
**Post date:** [26 January 2024 10:02 UTC](https://discourse.nodered.org/t/remove-maximum-number-of-catches/83187/18 "2024-01-26T10:02:10Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.