Using a watch node

General
1

Hello everyone.
In most of my flows I use a watch node to watch for excel files and read their content.
However, the watch nodes seem to work best if I cut and paste the file to the watched folder.
If I copy and paste, or drag it to the watched folder, I may receive an 'empty' output, or 'resource is busy / locked'.

How can this be resolved? I often find myself pasting a file into the watched folder a couple of times until it finally succeeds to read the objects within.

Thanks.

2

....interfacing with files that are created via a remote connection are suffering from these sync issues, depending on transfer protocols used.

The best I'd like to offer is the advice to actually not watch for the file with the required content, but rather configure the node to watch for a "semaphore" file instead.

You (or a remote system) can create that file at a time, when the intend is clear (the file with the real data/content has been physically provided/written) to start the flow/trigger the watch node.
After that event, use another node to read the data-file and delete the semaphore file via the flow.

Edit: I prefer to keep the semaphore file as small as possible and make this hold the path and name of the data file it triggers.

3

Thanks for the advice, any chance you could show an example?

4

not in NR atm, no.
But it should work as you currently do, just ad a sequence....
Example pattern:

data-file: "my-data.csv"
sem-file: "my-data.csv.OK"
Let the watch node watch for the .OK pattern.
then add a node in the flow, that reads the data file.
The name of the data-file is either the filename of the sem-file, without the trailing .OK or even written sidie the sem-file, which gets transported as a message from the watch node to your node, that reads the real data-file.

5

I've mentioned this before but the watch node does not expose all of the features of the underlying library in use Chokidar. If you fancy giving your JavaScript skills a bashing, you can actually use that library direct in a function node. Chokidar has a number of advanced settings that let you fine tune it including one the stops you from getting the notification you are referring to.

Sorry, but rather short on time at the mo so I can't really do you a demo.

6

Are you watching the folder or individual named files?
If watching the folder then I suspect you are hitting the problem that the Watch node can repeatedly trigger while a file is being written (as it doesn't know when the write is complete). To get around this add a Trigger node configured like this and it will output 5 seconds after the file stops being written to, which should guarantee that the write is complete. Make sure you get all the settings correct, including the Handling field

image
image517Ă—605 33.3 KB

7

Thanks Colin, ill give it a go in a couple of my flows and see how it handles things.

8

I started filling out a feature request for the Watch node to expose the awaitWriteFinish option in chokidar. However, when I run
sudo npm list -g chokidar
it does not seem to be installed. I would have expected to see it if the watch node uses it. Am I missing something?

9

..this only helps for files that arrive via a local protocol. When the file is transferred via a remote session, like via ftp and the underlying connection drops, the file will appear as closed/writeFinished, doesn't it?

10

No, npm packages don't quite work like that. You can look at the code for the node to see that Chokidar is a dependency. If you want to play with it in a function node, simply add it as a dependency in the setup of the node. It will be installed to your userDir automatically.

11

OK, I thought I understood how that worked. I assumed that chokidar would have to be installed as part of the node-red install in order for it to be used.

I don't think I need to try it, the operation seems fairly clear from the docs. Assuming I am right then it should be possible to use awaitWriteFinish in order to remove the virtually compulsory use of a trigger node after it.

1 Like
12

Sorry, I don't understand what you are saying.

13

...you were referring to the awaitWriteFinish option of the underlying lib of the watch node.
Using this would enable the node to detect if a file is still in use (being transferred/written to) from another application, hence preventing it to open/forward the file before it is completely written.

I was trying to say, that my experience with this kind of access control for syncing file access is not of much use, if the client is using a remote transfer protocol, i.e ftp.
Should the ftp session drop (i.e. due to a unstable connection) the ftp-server would still close the file, even when its content is not completely transferred and the watch node would still start catchinbg that incomplete file.
IMHO using this option method is only good for files where both sides, client/writer and consumer have access to the local file-system service / local OS.

1 Like
14

That's right. In remote cases, it is better to use a proper file transfer protocol that tells the client when the transfer is complete.

15

Yes, you are right, but it is still worth doing as that covers the majority of uses, and in particular would cover the requirement of this thread.

16

...actually I originally thought that the OP would not run NR and Excel-client on the same host, so some kind of remote transfer would have to occur. I also found SMB to be unreliable in that respect as well, so using a remote, network folder might not be good enough for that purpose.

17

It is possible that you are right, but there is nothing in the post to suggest that, as far as I can see.

Yes, watch does not work well over remote connections.

1 Like
18

I have raised a feature request. Watch node - add option to wait till file write completion

Wow, post 59999. That is a lot of posts.

1 Like
19

Can you point me at the code please Julian, perhaps I am looking in the wrong place. I can't see any reference to chokidar in node-red/packages/node_modules/@node-red/nodes/core/storage/23-watch.js. It seems to be using fs.notify.

20

Drat, I'm sure that I'd seen chokidar in there - I blame it on age (rather than my own stupidity!).

There are a couple of custom nodes that do use chokidar. Or, as mentioned, you can roll your own. I did do a quick test the other day and got it working in a function node. The trick seems to be getting rid of the watch when Node-RED re-deploys the node.

Chokidar is the library used by nearly all tools and services that do serious filing system watches. Like PM2 and many of the build tools (gulp, etc) and front-end framework dev servers (REACT, Vue, etc).

Just had another look at the Node-RED source. I actually found the reference in the package-lock.json so chokidar is used in a node-red dependency somewhere rather than by node-red itself.