Hi, before raising an issue, I'd like to gather more specific information for being able to provide useful information to the bug-fixing process.
What is happening: I'm using the email in node to poll every 10 seconds an Exchange Online mailbox. Every now and then (every few days, sometimes twice a day) the fetching process hangs and the email in node is stuck at the connecting status. The only way out of that (at least I know) is, to restart the NR flows. Then, it works for a while, but, it happens all over again, recently, several times a day.
@dceejay, BTW, that's why I needed the done. However, I've implemented a timeout scenario by using your recommendation with the status node to catch the hanging email node.
So, here's my question: is there a way to gather more information for a particular flow or node?
Well knowing that it is during the connection phase is a help. ( though every 10 s does sound like hammering the server to me.) Did you try reducing the frequency at all ?
I didn’t notice any difference when I’m polling every 30 secs.
I understand that it’s high-frequency. The reason why I’m doing this is, because a surveillance camera sends an email with a picture and I have to respond near real-time. The camera model doesn’t provide any other interface.
You are almost certainly getting a server connection timeout. Mail servers don't expect to be connected to even every minute let alone every 10s. It isn't that uncommon for email server connections to take >5 sec. I think that I have outlook set to server timeouts of a minute on the two enterprise accounts that I use for work. You are lucky that the server isn't blacklisting you.
Even a 5 minute connection would be considered very rapid. 10 minutes to an hour is better. If you need more, you need to find something that supports one of Exchange's push/sync methods.
As I said, 30s is still very short and you might still get a timeout from a busy server. I can't really say without understanding the way that the Exchange server(s) are set up and without knowing how busy they might be - or indeed, how resource constrained.
Also note that if this is an enterprise configuration of Exchange, it is common to handle load by building "out" rather than "up". That means that you end up with a load of Exchange servers and load balancing. Again, it is very common to find one or more of those servers misbehaving, badly configured or with some other oddity happening (it's Windows after all!). So you will commonly see apparently random client issues that are actually caused by those servers and none of the others.
In your instance, it would almost certainly be better to create a service that used a proper Exchange connection supporting push or ActiveSync as an intermediary. Potentially you could also do something clever with a server-side filter script though I'm not sure how fast they respond.
By the way, if this is an enterprise Exchange setup, I mentioned blacklisting and you should be careful since a front-end threat management gateway or intrusion protection system might well see your connections as an attack and auto-block. Obviously not happening to you at the moment but upgrades to infrastructure might easily introduce such checks especially if anyone were to audit the connection logs.
@TotallyInformation, I do understand your concerns; However, since the server is an Office 365 SaaS I'm not able to do any server scripting. Creating an ActiveSync service is also beyond my capabilities 
Nevertheless, if the server is running into a timeout, shouldn't the email node return with an error?
I have adjusted the timeout handling slightly. It was set the same as the retry time so there may have been a slight possibility that it could have missed the error if the previous attempt finished and the handler was destroyed before the error callback was called.
I'd be surprised but... none the less I have shortened the timeout fractionally so the timeout should always occur and flag the status error. (and then retry).
Will push to npm in a minute.
Thanks, @dceejay. I appreciate your responsiveness!
It is built-in, it is somewhat limited but if you go into Outlook Web, have a look at filters. Doubtless would need some clever lateral thinking to get something out of it - but you never know, you might think of something. Depending on your Office 365 licensing, you might also have access to Power Apps or Flow and that is reasonably powerful and you might be able to create a flow that would post to a Node-RED http-in node, using it as a "webhook".
1 Like
@dceejay, sorry, for the delayed reply. I noticed the following behavior:
- The
email node switches immediately to status connect error
- To make sure it indeed doesn't connect, I changed the protocol from
IMAP to POP3 (both SSL); it does connect and fetches the emails. Unfortunately, this doesn't help my purpose, I need IMAP to access the mailbox folders
- Nevertheless, even if the node connects and fetches, the status remains to
connect error
- (sending emails with
email out works fine with the same mailbox)
Please, tell me how I can help to identify the issue
Thanks!
1.7.7 works again. I will observe and keep you posted.
Thanks!
1 Like
You could always use a email gateway service to push notifications.
I use Pushover, who will give you a custom 'pushover' email address, such as dfx6gxt443@pomail.net which you could add to the camera interface.
Pushover will then very quickly convert that to a push message, and push it to node-red-node-pushover (as well as pushover's mobile app).
All this happens really quickly - 2 or 3 seconds, and you can also select the alert priority.
I've used this in a flow for the past 3 years for urgent alerts - it's reliable, really fast, and less than 7,500 alerts per month are free.
1 Like
Very interesting advice, @Paul-Reed, thanks. I have had a look at Pushover, and although the service would be doing exactly what I need, its commercial component is disappointing. 7500 a month would be about 10 messages an hour. I can achieve that volume with a simple email as well. The next step would be 10,000 for $50, which is, in that case, too much.
I also have had a look at what @TotallyInformation suggested. The idea of looking into Microsoft Flow or PowerApps was great, although I couldn't find an appropriate connector to something outside the Microsoft world (eg, like IFTTT or Telegram).
Of course exposing an own webhook to Microsoft Push service would be the ideal scenario, but that would be beyond my coding capabilities.
Thanks anyway!
Although I haven't tried it, there appears to be an email gateway for Telegram - https://botostore.com/c/email2telegrambot/
I believe that Telegram allows 1 message per second before hitting limits.
Why do you need email to push notifications again?
I use Telegram which replaced pushover for me due both to their limits and the way that the clients worked. Telegram has by far the easiest bot interface I've worked with and it is very reliable, rarely going over a lag of 1-2 seconds and usually much less.
By the way, if you are looking to process emails, I might also suggest looking at imapfilter which is a command line IMAP management tool. Now I have my new "home server" (Lenovo ThinkPad x240 laptop), I've enough headroom to play with more things and imapfilter was top of my list in order to try and take back more control over my various mailboxes. It is very efficient and, because it is written in LUA, it is easy to interface with Node-RED. Here is a quick test where each time imapfilter runs (there is also a daemon mode if your IMAP server supports push) I send some rubbish data to Node-RED via an http-in/-out pair of nodes.
-- Execute a shell script and capture the output
local function osExecute(cmd)
local fileHandle = assert(io.popen(cmd, 'r'))
local commandOutput = assert(fileHandle:read('*a'))
local returnTable = {fileHandle:close()}
return commandOutput,returnTable -- [3] -- rc[3] contains returnCode
end
output, tbl = osExecute('curl "http://localhost:1880/mailinfo?wibble=true&wobble=40" --silent --connect-timeout 0.1 --max-time 0.5')
print('OUTPUT: ', output)
print('TBL: ', dump(tbl))
Obviously, you would normally connect to your mail server, get some info which could be stats or whole emails and feed that to Node-RED.
There are always different ways of doing things.
I’m not sure if imapfilter would solve my problem. But, perhaps I cannot see the forest for the trees. Let me recap what I’m trying to achieve:
A surveillance camera sends emails with an attached picture as soon as movement is detected. The camera has no other interface (like ONVIF or API or other).
The information the email contains shall be transmitted to a telegram bot as fast as possible. For doing so, I’m fetching the email from the particular inbox folder, process it and send it to the telegram chat.
The flow works, the issue was that the email node used to “hang” now and then, and I had to restart the node-red server to start the flow again.
So, if fetching the emails is bad, I’d be happy to achieve the same result via any other way.
Well you could certainly do this via imapfilter. However, the main issue is one of timing. If your IMAP server supports IMAP push (IMAP NOTIFY), then you can configure imapfilter to stay alive in the background and it will start processing when the server signifies a new email has arrived.
If you don't mind running once a minute lets say, then probably running imapfilter via cron is probably more reliable maybe, not sure. Much less than that and you might run the risk of your mail server tagging you as a hacker process and blocking you.
Of course, if you have control over your mailserver, there might be lots of other alternatives. If you have access to the mail server, you could add a call out to your own script directly into the server in the same way that seive or spamassassin runs. Or if you are using something like the venerable procmail, its built-in scripting is probably enough.
If you do use imapfilter, note that it is able to read the whole email if you need it to and you could use LUA to process that into something meaningful for Node-RED to work with. You can also mark the email read so you don't process it twice and can set flags on it too if you need to (or delete it).
It is very powerful and I'm only scratching the surface though the config file that processes my main inbox is already nearly 300 lines long even so it generally runs in a couple of seconds at most. (It is a catch-all account for the domain as well so it gets a lot of rubbish).
I'm dealing with Exchange Online (Outlook 365), hence, I have no access to the email server at all.
However, I took another path. @TotallyInformation's hint regarding PowerApps was a good one.
Microsoft Flow takes advantage of Exchange Online push notifications. So, I created a flow that writes the email attachment to DropBox every time a new email arrives in the specific folder:
In NodeRED, with the help of
node-red-node-dropbox I check for a new file in the DropBox folder, download it and send it to the Telegram chat:
It works like a charm!
Thanks to all of you for your contributions and the many creative ideas!
2 Likes
Hi ,
I have basicly the same issue with EmailIn node stuck in connecting status.
I am polling ISP email server every 2 minutes.
It works ok for many days up to few weeks but then locks.
As recommended I have lower frequency to 5 minutes and I have installed the latest
version 1.7.7 .
This was done today
