Filtering an array

General
1

Is there a way to filter old entries from an array and only keep the new entries?

I have a flow that gets forum posts as an array. The array is then passed through filter node so only when array contains a new element it goes through the filter node. But the problem is that the whole array is passed forward. Which means that the next node that sends a notification for each post in an array ends up re-sending notifications for old entries.

One solution I can think of:

  1. Add a function node after the filter node.
  2. Within that function node use context.set(); to store the previous version of the array.
  3. Compare the new array to the one stored in context and filter out the old entries then pass this new filtered array forward to the notification sender.

Is that the right solution or is there a better/optimal way to achieve this?

2

It is difficult to answer as we have no idea how your array is constructed, is it an array of strings, or objects. If objects is there a time property. Best supply an example of this array.

3

It is an array with strings only. No timestamps or anything else that can be used to determine if the entry is "old" or not.

4

Then you will have to keep the old version of the array to compare to the new and filter out everything from the old.

There are node.js packages that will do this for you and you can use one with function nodes quite easily now in node-red.

Check out my wiser repository on GitHub for one of the best of the diff processing packages. I can't remember the name off the top of my head.

5

You should be able to use a change node or a function node to filter on a stored previous version of the array.
e.g.

[{"id":"94fbbdc.243fd4","type":"inject","z":"c74669a0.6a34f8","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"[\"1\",\"2\",\"3\"]","payloadType":"json","x":140,"y":2780,"wires":[["c6d49d4e.cdc448"]]},{"id":"c6d49d4e.cdc448","type":"change","z":"c74669a0.6a34f8","name":"","rules":[{"t":"set","p":"hold","pt":"msg","to":"[$$.payload[$not($ in $flowContext(\"storeArray\"))]]","tot":"jsonata"},{"t":"set","p":"storeArray","pt":"flow","to":"payload","tot":"msg"},{"t":"move","p":"hold","pt":"msg","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":380,"y":2820,"wires":[["6e83ede8.d5fcd4"]]},{"id":"c73d9918.369ea","type":"inject","z":"c74669a0.6a34f8","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"[\"1\",\"2\",\"3\",\"4\"]","payloadType":"json","x":140,"y":2840,"wires":[["c6d49d4e.cdc448"]]},{"id":"6e83ede8.d5fcd4","type":"debug","z":"c74669a0.6a34f8","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":620,"y":2820,"wires":[]}]
1 Like
6

Is the array you get each time the total forum thread? If so then all you need to do is to keep track of how many you have already handled and drop that many off the front of the array.

7

Thanks for all the suggestions. So on top of the solution I suggested it seems this can also be done in few other ways:

  1. Probably the easiest one is to use a node package that does this.
  2. Another option is to use a change node. This is an interesting solution, but I feel for my specific use case it is a bit too complex compared to a function node that uses context to store the previous array.
  3. Same as my suggested solution, but instead of storing the full array you only store the number of items in previous array and then drop that number of entries from the beginning of the new array. (this works only if array keeps growing indefinitely. In my case the list grows only until a new page is "turned" in the forum thread)

I personally have no idea which of these solutions is the most efficient.

8

Don't worry about efficiency. Premature optimisation is a waste of time. Do it whichever seems easiest to you. If the performance of this bit of code becomes an issue (which it won't) then you can optimise it at that point.

2 Likes
9

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.