I've lost everything. CONNECTION LOST TO MACHINE error - subflow connections

[Trying_to_learn]

And I don't think there is any going back.

I did backup the files, but alas they were in the same directory and so NODE-RED wiped the backups I made.

I've lost all my sub-flows......

All my tabs (there was about 30 of them).

Long post.

I am seeing this error coming up a bit these days from other people too.

I am on a NUC - intel i7 I think. It has power. It isn't a RasPi.

It has memory.

Some of the tabs are "disabled" and some aren't. (That's life)

This also ties in with the machine's load.

I'm using FF (on all machines) but the RasPies are all headless.

Now and then when I am editing flows on ANY machine, it just locks up.

I have a "performance" tab and it does some wild stuff.

I'll get to that later. (maybe a later post).

I saw that NR had made .backup files of my stuff.

I coppied them to a file name -BACKUP to make it even more different.
Wiped.

I won't do anything with Node-Red just now.
I've also disabled it sudo systemctl disable nodered.service

On an off chance: Can I get those files back?

---

The problem/s that are/were happening.

When modifying flows on ANY machine, it would just lock up (I know I have said this, but to keep things altogether).
The performance tab would show loads ranging from 20 --> thousands (sometimes) but maybe 70's 80's.

Fluctuations all over the place.

After some time it would settle down and things would be normal.

Two things I have learnt:
1 - if I open top (hold on!) and look I would see the CPU load at (or greater than) 100%.
There would be Node-red listed and now and then web-services (or something like that) listed showing massive load.
2 - it wasn't always on the machine of interest. Though mostly I think it was.

(Yeah, this isn't helping me really)

I have a few foreign nodes installed as they seem to help me get things done.
No offence to anyone for what I am about to say. It is my best recollection of things.
WAY way back, I added ui_level.

Shortly there after that machine would lock up as per what is happening now.

I have a few more nodes like that and I am slightly suspicious there is something going on with them.
Though I did mention it at the time and tried to give as much information, I could never prove anything.

Moving on:
Another thing I learnt was this web-services thing.....
It was really stumping me.
Yes, I had some fancy dashboards, but it just didn't feel right.

The other day I found the problem.
That entry web-services hogging 99% of the CPU load, and there were multiple lines of it too!
That is/was a loop in a flow.
Luckily I found it fairly quickly with help of my other thread about message counting and frequency.
This thread
I found the problem.

The frequency of message parsing was > 600 per second.

I would try to do things and NR was just dead in the water.
After several things, I stated it in --safe mode and added the code to catch that error.

So: where's all this going?

Well,
That loop generated these web-services errors and all that.
Though I am not exactly seeing that now, I am wondering if all the flows I have are spiking the load of NR at times and causing this lag?

I'd better post this as I am really stuck and would prefer to get things back to normal.
(Yeah, back up. I did. I just didn't realise NR wiped it's own directory as much as it does.)

If you are still reading:
I would like to get my old files back if at all possible.

Sorry if I left this a bit in the air. I wanted to get it out there ASAP to get help, but also wanted to share what I have recently seen. I'll get back to my theories when I calm down more than I am just now.

[knolleary]

The .backup files are the previous versions of the files - created whenever you deploy a new version. They are not a backup of the latest version.

Node-RED doesn't create multiple copies of the flow files. You need to be backing them up yourself.

If you have deleted all the files, and have no backups, then there isn't a lot we can do.

[Trying_to_learn]

I was more thinking at an OS level.

I made copies of the files but they were in the NR directory.

If they were deleted, is there a way to get them back at the OS level?

(yes outside the forums area, but I should ask)

[Trying_to_learn]

I'm back home now. (Was out buying the necessities.)

Not assigning blame, but more on/of the story:

The story - more of it.

I was playing around with a flow wanting to tweak it.
This is on a RasPi and I didn't want to bother it with all the testing, so I decided to do it on the NUC.

I needed to use my timestamp node (subflow).
I dragged it into the edit area, wrote the flow and deployed.
Pressed the button and: (drum roll) nothing.

"That's weird"

Shortly there after I opened the subflow and: (another drum roll)
All the connections from the IN/OUT to the other nodes were missing.
All the other internal (node to node) connections existed.

Ok, two things:
1 - I hadn't fixed that subflow on THIS machine. Though I had fixed it on all the remote machines, because they use it, and it is working.
2 - The problem still exists.

I can't (now) say. It's all gone-ski.

I connected the wires and deployed it.
Waiting waiting waiting.

Connection lost to server. (or what ever that message is)

This is a red flag to me (well, a flag of some colour, as it is being mentioned a lot recently in the forum) as it has been happening a fair bit to me of late.
I've just (usually waited).

Stupid me, I didn't this time.

That is now water under the bridge. I've lost a lot of work. Such is.

I don't know if I lost all the installed nodes or not. I'll find out when I am confident enough to get NR running again on this machine.

However, this CONNECTION LOST TO SERVER is definitely annoying, and I too am seeing it too.

I mentioned how I discovered that loops in the code lock up the machine and how I did some tracing on it to prove that is correct.

I'll stop now, as I think I have said enough for now to explain the story better than I did with the first post.

For the sake of showing what I see here are some examples that I see for the load on the web page after I simply pressed an inject node a few times to list events.

And the screen shot which started this whole lot of hurt for me.

[edje11]

Maybe a bit to late now for you.
I'm backing up my flows twice a week automatic to Dropbox with the dsm node and this script.
Then you have a copy outside the house and it's saved by date so you can always go back.

[Trying_to_learn]

Yeah. A bit late.

I have a script I use on other remote machines and I have now got it working on this machine.

Not weekly, or anything that elaborate.

It is just a script I can run when I know I am going to be doing anything "dangerous".

[ghayne]

@Trying_to_learn

It is just a script I can run when I know I am going to be doing anything "dangerous".

Have you never considered using the Projects option?

[Trying_to_learn]

projects option.

What's that?

(Yeah, I ask dumb questions)

[ghayne]

https://nodered.org/docs/user-guide/projects/

[Trying_to_learn]

(I deserved that)

Yeah, it relies on the internet. Good if you live in a first world country.
I live in Oz.

Dial up is fast.

I have only just touched the git thing.

It is another thing I have to learn how to use.
I'm not sure I should do that if I can't get this part working good.

When I get time (and I know there is probably no excuse for that just now - globally) I may look into it.

[Paul-Reed]

Backing up to Dropbox is much easier than projects, also it can be run automatically so it's easy to fit & forget (until you need a backup).
I started off with projects, but got myself in all sorts of problems.
Unless you are confident with git, I'd leave it alone Andrew. (Just my personal opinion)

[Trying_to_learn]

Thanks Paul.

yeah, I think I shall leave well enough alone just now and use my script to back up the flows.
I know that is painful/wasteful as it backs up everything.

But I guess: Nothing is perfect. I just need to use what I find best.

Though what I may also do is export big blocks of code to files as I have done anyway in the past.

I may need to get into this GIT thing more one day... But I think baby steps at this time.
Thanks.

[Colin]

@Trying_to_learn

Yeah, it relies on the internet .

No it doesn't, you can have the repository on the same machine or on another machine on your network.

[Trying_to_learn]

Oh. So (dumb question) why do I need keys?

(From the link I was shown)

The feature relies on having the git and ssh-keygen command line tools available. Node-RED will check for them on start-up and let you know if they are missing.

As I said, I think I had maybe better leave it alone.

When we can venture out again (when ever that is) I may have to ask for a 1 on 1 from someone who can talk the talk, and walk the walk and get them to show me what it all is about.