RSS Feed - only get the latest entry

General
1

Noob question. I am using NodeRed to read youtube rss and post the results to chat bot. I have noticed that when the node red server goes offline or gets rebooted the entire feed going back many entries gets reposted.

question:
Would it be better to somehow record everything posted to the database and then compare new entries with DB? That seems somewhat complex.

Or can I just tune the current setup to only post content posted to RSS feed in the last hour? That way if system does restart the number of duplicates will be low. If so what would be the best way to do it?

2

Welcome to the forum @pitheway

I don't use rss feeds, but would it be enough to record the timestamp of the latest one actioned? If so you could use persistent context in node red to remember that so you can start again from there on restart. There is a node red docs page "Working with Context" that tells you about that.

Have you watched the node red essentials videos (they would have covered context if I remember correctly). Node-RED Essentials. Well worth watching if you have not done so.

3

The context is a good idea!

contextStorage: {
default: "memoryOnly",
memoryOnly: { module: 'memory' },
file: { module: 'localfilesystem' }
},

This is what I have in settings,js. Reset the service but still dont see context in change node. Any clues what I missed?

4

Did you refresh the browser after restarting node-red?

You should see this in a change node...
image

Your settings entry is slightly different to mine but it should work. You could try what I have which is...

    contextStorage: {
        default: { module: "memory" },
        file: { module: "localfilesystem" }
    },

PS: what version of node-red are you running?

5

Show us how you are saving the value to context and how you are picking it up again.

6

I rebooted the pi and now I can see the drop down in the change node. Looks like it only appears when flow or global is selected.

7

Context stores only apply to context.

8

Thats good to know. I now see the youtube link that should be the payload in context window.

So how would this work in practice?

I have a function node that translates payload to telegram message:

var content = msg.payload
msg.payload = {}
msg.payload.chatId = 11111111
msg.payload.type = "message"
msg.payload.content = content
return msg;

However when I use flow.payload the content part of the message is null. Are there some other changes I need to make?

Here is the flow:

[{"id":"e7aba38c.4139d","type":"tab","label":"Flow 5","disabled":false,"info":""},{"id":"9481a6af.3aa9d8","type":"function","z":"e7aba38c.4139d","name":"","func":"var content = flow.payload\nmsg.payload = {}\nmsg.payload.chatId = 11111111\nmsg.payload.type = "message"\nmsg.payload.content = content\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":,"x":840,"y":360,"wires":[["6bcefcbc.f42cf4"]]},{"id":"6bcefcbc.f42cf4","type":"debug","z":"e7aba38c.4139d","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1090,"y":360,"wires":},{"id":"a7d9d8db.481e58","type":"change","z":"e7aba38c.4139d","name":"","rules":[{"t":"move","p":"topic","pt":"msg","to":"#:(file)::payload","tot":"flow"},{"t":"set","p":"topic","pt":"msg","to":"article.author & " - " & article.title","tot":"jsonata"},{"t":"set","p":"image","pt":"msg","to":"article.image.url","tot":"msg"},{"t":"delete","p":"article","pt":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":340,"y":360,"wires":[["fac2988.672b568"]]},{"id":"fac2988.672b568","type":"rbe","z":"e7aba38c.4139d","name":"","func":"rbe","gap":"","start":"","inout":"out","septopics":true,"property":"payload","x":590,"y":360,"wires":[["9481a6af.3aa9d8"]]},{"id":"e5347985.f13dc8","type":"feedparse","z":"e7aba38c.4139d","name":"feed","url":"https://www.youtube.com/feeds/videos.xml?channel_id=UCu94OHbBYVUXLYPh4NBu10w","interval":"2","x":110,"y":360,"wires":[["a7d9d8db.481e58"]]}]

Thank you all for taking time to answer my ridiculous questions.

9

In order to make code readable and usable it is necessary to surround your code with three backticks (also known as a left quote or backquote ```)

``` 
   code goes here 
```

You can edit and correct your post by clicking the pencil :pencil2: icon.

See this post for more details - How to share code or flow json

11

Even without it being correctly formatted, I can see that the function is not the same as you showed above. The code you posted says
var content = msg.payload
but in the function you have
var content = flow.payload
That is not how to fetch context data in a function node, you have to use flow.get(). In Working with context : Node-RED it has a section Using context in a function node, which links to Writing Functions : Node-RED where it shows you how to do that.

Also, though, the value it picks up will be the value it wrote there a in the Change node earlier in the flow. I thought you wanted to remember something each time through, then the first time on restart pick that up and use that to know where to start, and I don't see how your flow does that. What is it that you are writing to context?

12

var content = flow.payload - this was me trying to get the payload into telegram message. And yes I do want to remember the entire history of rss.
Like I said I am very new to this. Hence was hoping for some assistance. Perhaps not just functional but also logical as well.
maybe context is the way to do it.

13

Exactly what do you need to remember? Not in vague terms, but exactly what data.

14

I would like to remember the payload from the change that follows RSS node that in this case contains the link to you tube video. And then potentially use the RBE or some other method to prevent repeat messages.

I now see how this may become complicated quickly since I will need to pull only that part in the subsequent telegram node...

15

Do you need to record all of them since the beginning or just the latest one? When fetching from a feed how does it know which ones you have already got? Do they not have a timestamp or a sequence number or something?

16

I would like to keep at least a few days worth to verify against. The feed gets everything but the RBE node prevents duplicates. And that works well until the reboot

I have not looked at pulling the timestamp though its a good idea.

17

If this is possible then it is a good way to do it, so I suggest you investigate that.

18

OK and then somehow pull the link from context using the timestamp?

19

Is it possible to ask for videos posted after a certain time?
Does each video you receive have a timestamp with it?
Which RSS node are you using?

20

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.