I have one application in which a Linux machine capturing and saving to RAM disk a lot of traffic (telemetry). This traffic is quite huge - about 700MB/s. I do it with TCPdump, which works just fine. It creates a sequentially numbered files, which have 2GB size and have a Wireshark PCAP format. Older files are either archived or overwritten.
I would like to make a kind of lightweight monitoring app, which would just open the latest captured file, browse through few first captured frames, extract the data from them and show it on the dashboard. This has to be done periodically, let's say every couple of seconds.
Off-course NodeRED might be not the best tool for it, but due to next ideas I would like to start with it - thus the question - is it possible to do with NodeRED? Can it open large files quickly, without reading them fully?
To answer the original question... yes it does handle streaming of files - but no it doesn't have a mechanism to just read the first few lines. So net is no don't use the file node.
You could as suggested look at directly ingesting tcpdump output (or maybe tshark) - or you could maybe use external filter like the head command to just read the first part of a file and then just process that.
I think tshark is a pretty much the same tool as tcpdump. At least it uses the same set of libraries and options...
Well, the only way to filter it - is to capture a limited amount of frames - let's say - 20 and stop. Then NodeRed should read the file (our output of TCPdump/TShark) and show it in the dashboard. This should be done by exec node in periodical basis.
Not sure if it's possible on the same machine, but I can give it a try.
To make understanding a bit easier (for me and for others) - imagine that this is an MPEG video stream from the IP camera with the rate of 700Mb/s and frame rate of about 10000 frames per second. Simply as that. And I have a kind of network video recorder which records this stream continuously to NVMe storage.
In addition to this I just want to grab one frame out of this stream every second and show it on the screen. The accurate time stamp of the frame doesn't matter and it shouldn't be the perfect timing - it's just a simple frame grabber for monitoring purposes.
My real stream is not a video, but concept is absolutely same.
My data stream consists of the periodical MAC frames of the same type, protocol, source, destination etc. The only difference between them is in payload. Payload consists of some monotonically increasing counter and the rest are values, which need to be shown in monitoring app. Therefore all I can is to capture every nth frame if I can do this based on counter value, but most likely it's impossible.