How good is NodeRED in reading files?

General
1

Hi all,

I have one application in which a Linux machine capturing and saving to RAM disk a lot of traffic (telemetry). This traffic is quite huge - about 700MB/s. I do it with TCPdump, which works just fine. It creates a sequentially numbered files, which have 2GB size and have a Wireshark PCAP format. Older files are either archived or overwritten.

I would like to make a kind of lightweight monitoring app, which would just open the latest captured file, browse through few first captured frames, extract the data from them and show it on the dashboard. This has to be done periodically, let's say every couple of seconds.

Off-course NodeRED might be not the best tool for it, but due to next ideas I would like to start with it - thus the question - is it possible to do with NodeRED? Can it open large files quickly, without reading them fully?

1 Like
2

That is a lot of data to process in a matter of seconds.

There is of course fs.createReadStream in nodejs so it is feasible but I am unsure how practical it is.

Could your wireshark capture be filtered in anyway to reduce data?

Perhaps there is also an option to send wireshark direct - perhaps TCP - to your node-red (cutting out file)?

3

This is another option I'm thinking of. This is not a TCP or UDP traffic - it's just a broadcasted MAC frames stream, which might be capturable by NodeRED direct, but I didn't check this yet.

4

You could possibly use tcpdump from an exec or daemon node

5

It will also depend on the device running Node-RED. The early RPI Model 2 only had 256MB of memory and operates at 700 MHz while my MacMini (late 2012) has 8 GB memory and runs at 2.5 GHz.

6

Instead of wireshark, you could use tshark, which is the commandline version

https://www.wireshark.org/docs/man-pages/tshark.html

7

To answer the original question... yes it does handle streaming of files - but no it doesn't have a mechanism to just read the first few lines. So net is no don't use the file node.
You could as suggested look at directly ingesting tcpdump output (or maybe tshark) - or you could maybe use external filter like the head command to just read the first part of a file and then just process that.

8

I think tshark is a pretty much the same tool as tcpdump. At least it uses the same set of libraries and options...

Well, the only way to filter it - is to capture a limited amount of frames - let's say - 20 and stop. Then NodeRed should read the file (our output of TCPdump/TShark) and show it in the dashboard. This should be done by exec node in periodical basis.
Not sure if it's possible on the same machine, but I can give it a try.

9

I was referring to setting flags on the command line to TCPDump (e.g. tcpdump -ntqr)

Is there no way you can reduce the volume of data at source? (instead of trying to parse 700MB of data was really my point).

10

No, there is no way. It's just a data stream.

To make understanding a bit easier (for me and for others) - imagine that this is an MPEG video stream from the IP camera with the rate of 700Mb/s and frame rate of about 10000 frames per second. Simply as that. And I have a kind of network video recorder which records this stream continuously to NVMe storage.
In addition to this I just want to grab one frame out of this stream every second and show it on the screen. The accurate time stamp of the frame doesn't matter and it shouldn't be the perfect timing - it's just a simple frame grabber for monitoring purposes.

My real stream is not a video, but concept is absolutely same.

11

I understand the concept.

Let me try one last time.

Do you need everything that flies across the Ethernet port? you can filter out things that are not if interest e.g. tcpdump -i any icmp only gets icmp data

So unless you have a need to capture everything, then perhaps consider a filter on TCPDump

Reference

Lastly, as I re-read your post

I now think @dceejay suggestion of using head might be a reasonably simple solution Reference

12

My data stream consists of the periodical MAC frames of the same type, protocol, source, destination etc. The only difference between them is in payload. Payload consists of some monotonically increasing counter and the rest are values, which need to be shown in monitoring app. Therefore all I can is to capture every nth frame if I can do this based on counter value, but most likely it's impossible.

I will check head solution. Thanks

13

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.