After upgrade to v3, json errors in log

General
1

Today I upgraded my node-red main install from v2.2.2 to v3.0.2. As I didn't have success by updating the existing install, I decided to bring up a new one.

So I created a new lxc container with the new version.
Edited the settings.json file to activate context storage and a couple of required env variables.
Manually installed the required nodes with npm.
Then transferred files/folders from the old install to the new with:
scp -r root@10.1.5.42:.node-red/{flows.json,flows_cred.json,context\} .
I wasn't using credential encryption in the previous install.

It all seemed to work ok, nodered started with all the flows and nodes, and it seems to work.
But (asides from some recurring modbus coms errors), the following json is periodically coming to the log:

12 Mar 11:16:53 - [error] [modbus-getter:Grid] Error: Timed out
    at /root/.node-red/node_modules/node-red-contrib-modbus/modbus/maps/core/core/modbus-client-core.js:81:15
{
  id: '85c9e660be57225d',
  type: 'spline-curve',
  z: '7efd98b203faf491',
  name: 'daylight curve',
  output_key: 'brightness',
  input_key: 'payload.times_of_day.active.perc',
  points: [
    { x: 0, y: 0.744, uid: 0, index: 0 },
    { x: 0.03, y: 0.838, uid: 1, index: 0 },
    { x: 0.097, y: 0.941, uid: 2, index: 0 },
    { x: 0.183, y: 1, uid: 3, index: 0 },
    { x: 0.86, y: 0.995, uid: 4, index: 0 },
    { x: 0.943, y: 0.931, uid: 5, index: 0 },
    { x: 1, y: 0.85, uid: 6, index: 0 }
  ],
  x: 660,
  y: 280,
  wires: [ [ 'bfd318c3ce5ec3b5' ] ]
}
[ 'brightness' ]
{
  id: 'bfd318c3ce5ec3b5',
  type: 'spline-curve',
  z: '7efd98b203faf491',
  name: 'daylight curve',
  output_key: 'temperature',
  input_key: 'payload.times_of_day.active.perc',
  points: [
    { x: 0, y: 0.54, uid: 0, index: 0 },
    { x: 0.046, y: 0.744, uid: 1, index: 0 },
    { x: 0.096, y: 0.784, uid: 2, index: 0 },
    { x: 0.163, y: 0.797, uid: 3, index: 0 },
    { x: 0.22, y: 0.837, uid: 4, index: 0 },
    { x: 0.29, y: 0.924, uid: 5, index: 0 },
    { x: 0.376, y: 0.844, uid: 6, index: 0 },
    { x: 0.463, y: 0.8, uid: 7, index: 0 },
    { x: 0.9, y: 0.724, uid: 8, index: 0 },
    { x: 0.96, y: 0.694, uid: 9, index: 0 },
    { x: 1, y: 0.65, uid: 10, index: 0 }
  ],
  x: 840,
  y: 280,
  wires: [ [ 'e6d140eb3d63a405' ] ]
}
[ 'temperature' ]

note: The call stak showing on the first line seems to have nothing to do with the flow json showed. The json appears without any time mark.

Does anyone know what that could mean?
It seems to have something to do with the following flow. And also the error seems to appear with the inject node interval.

imagen
imagen1010Ă—378 54.8 KB

The debug console doesn't show anything though.

2

What node type are the spline nodes? node-red-contrib-something?

3

Assuming that it is node-red-contrib-spline-curve then I see that in August last year there was a commit added that removed logging of the config and output keys, which I suspect is what you are seeing. The node has not been released since then so the current release still has that code in. You could install it direct from github by going into your .node-red directory and running
npm install jcronq/node-red-contrib-spline-curve
and restarting node-red.

4

They are indeed node-red-contrib-spline-curve.
But I don't understand... it is already installed. I included it in the npm install line when I setup the node.

EDIT: Aahhh ok, I understand now. Sorry. You meant that I should install the node directly from the source repo. So those messages should had already been on my previous install.
Thank you very much Colin. :+1:
It seems that you're on this at full time.

5

You have installed it from npmjs which has version 1.1.8 dated three years ago. See node-red-contrib-spline-curve - npm.
The source code on github (GitHub - jcronq/node-red-contrib-spline-curve) has been updated since then, but the node has not been re-released. Using the command I posted you can install it direct from there.

For future reference you can save yourself a lot of time when moving a flow to an new system (or restoring from backup). If you copy the whole of the .node-red directory except the node_modules directory, go into .node-red and run
npm install
that will install all the extra nodes you previously had. The definition of those is contained in package.json and package-lock.json.

1 Like
6

I was afraid that this procedure you mentioned installed older versions of the nodes. Also differences that would exist in settings.json between versions.

Just one question on this: What would be for you the best way of keeping a full copy of the flows as a backup and also as a way to do this proceedure?
With flows I mean + required credentials, context if possible... all guts as if it was a docker install, but with also "source" control/versioning?

This week when I tried to upgrade from v2.2.2 it failed and I had to recover from the last full backup, that was from the past 5th. So I lost some small changes.

The solution would be syncing it to github? I think that would be bulkier and dangerous, isn't it?

PS: The solution worked and no more json in the logs.

7

Do you mean that the upgrade damaged the flows file? That seems very unlikely.

Backup the whole of the .node-red folder except for node_modules. I do this using git, committing after each significant update, using private repositories on github as a remote repo. I also run a daily routine backup of everything important using duplicity (there are many alternatives). I run that on each machine daily, backing up to a central server, and weekly to a remote server.

8

Do you mean using npm install? If you have restored package-lock.json then it will install exactly the versions that you previously had. If you delete package-lock.json then it will restore the latest versions. What to install is in package.json, what versions to install is in package-lock.json.

9

Would you mind sharing me how do you upload to git? Using manual commit and push commands? Automated with some script that detects changes or with a convenient button in the dashboard? :blush:

I run nodered in proxmox lxc container. And I do weekly backups. I should look for some way to go daily and incremental. I have too much pending tasks.

Nice.

10

Yes, using git gui usually.

11

I thought being you would use the projects. Why not?

12

It doesn't fit my workflow. Projects is designed for where you have multiple flow files that you want to run on one machine. I have multiple machines each with one flow file. To restore the setup on a machine I just have to install node-red, clone the appropriate repository, run npm install and start node-red.

1 Like
13

@Colin excuse me for bothering you again. I'd been searching at internet about how to do the dump to github and can't find details on it. I have some questions.

How do you organize the repo so it is cloned directly to the .nodered folder?
If I create a nodered-configuration private repo, when cloned, it will download to a nodered-configuration_master in the local machine. And if renamed it will stop working.

How to manage secrets, aka credentials?
I found all my credentials in the flows-cred.json in plain text, and the option for encrypting them is to have the key in settings.json, that I also would want to upload. Two nodes I use require configuration in there.

How do you manage git authentication? Do you generate a token and install it on your new machine?

14

When you clone it you can specify where to clone it to. I would not just call it nodered-configuration, I would make it more explicit in case you have more systems. So I have one for each of my Pis, called things like node-red_pi03. Then to clone it I do, on the pi,
git clone git@github.com:mygitusername/node-red_pi03 ~/.node-red

You can use an environment variable for storing the credential secret. Settings.js is just a javascript file so you can put any js code in there to pick it up. For example you can say
credentialSecret: process.env.MY_CREDENTIAL_SECRET

There are several options, I use ssh keys. These are my notes on how I did it.

Assuming you have already setup a key pair on the computer, ~/.ssh/id_rsa.pub, go to github acct and click the Profile button in top right and select Settings
Select ssh and GPG Keys in left pane.
New ssh Key (not GPG)
Give it a title (eg pi03 or mycomputername)
On pc: cat ~/.ssh/id_rsa.pub and copy/paste the whole string inc email address into the browser and click Add.
To test it
ssh -T git@github.com

15

Thanks Colin, you're great.

I didn't thought about specifying a destination for the clone. That's a little DOH.
But... how do you create that repo from an existing install with the standard naming? I suppose you do Git init + point to an existing repo so folder name gets overridden.

Then how you set this variable? Seen something about a config.json file or something simmilar..

16

I must admit, I do it a rather hacky way, but easy to do.
Create the new repository on github
Rename your .node-red folder temporarily
Clone the github repo to .node-red, which will make a new folder with just .git in it
Move all the files across from the original .node-red folder and commit them, and git push to send it up to github.
You will want something like this in .gitignore

*.backup*
node_modules
context
.config.users.json

.config.users.json has the editor runtime stuff in, like whether the right hand pane is open, which changes all the time, which is why you probably don't want to commit that.

That depends how you run node-red. If you use the systemd script you can add it to the script. Or you could add it to your system environment.
Or you could decide that actually your credentials aren't actually that critical and just rely on the github to keep it private, and put it in settings.js. It depends what the credentials are for and how serious it would be if they leaked out.

17

Ingenious. I thought one would prefer to also store the backups.

I'm running it as a service in systemd but it's a custom one as I use this repos for my deployment: Proxmox VE Helper-Scripts | Scripts for Streamlining Your Homelab with Proxmox VE
I think I'll use systemctl edit nodered.service to override and add my variable. That could be easily automated on deploy.

Well... if I was using a Telegram node, if compromised, someone could take possesion of the account. If I was using a Tesla node... someone could make my car do strange things while I'm driving, like control the MCU or Honk, open-close windows... The worst part would be that I suppose it could remotely engage driving or set valet mode and pick up the car.

Mmmmm.... :sweat_smile:

18

For me this is for source control rather than backup, as I have a separate backup strategy, so there isn't much point. It may be different for you.

Yes, that is the right way to do it. If you have an automated deploy I think you have have to place the override file in the appropriate location.

Yes indeed. As I said, it depends on what the credentials are.

19

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.