Node-RED CLI tool for easier source code management

Share Your Projects
1

The Node-RED editor is evolutionary but sometimes there is a need to manage a flow outside of it. I’m identifying the problem and proposing a command line tool to accommodate this; please share your thoughts.

Problem: You have a flow (json) file with multiple function and template nodes; you’d like to edit it in a traditional text editor or submit it to source code version control (such as a local git, svn, GitHub, GitLab, etc.) and:

  • View line by line differences using online comparison/differential tools
  • Edit locally using an IDE such as Visual Code (and benefit from all their sweet features; Co-Pilot, code folding, syntax highlighting for the content type, etc.)
  • Edit online via GitHub/Gitea/GitLab’s online tools.
  • See function and template node changes as manageable files (vs navigating one very long json file).
  • Use a simple command line editor.

The aforementioned is virtually impossible because a flow file (while convenient and compact) is one large file; even with JSON “pretty print” the function and template nodes are escape encoded. Identifying line by line changes is unreasonable outside of Node-RED. I’d propose the creation of two CLI tools to address this issue:

  1. flow2src - a command line tool you use to pass a parameter containing the file name of your flow file. Ie. “flow2src myflow.json”, which will in turn do the following:
  • Reads the myflow.json file and creates a folder of the same name.
  • Creates a file within the folder called “flow.json”, containing the original json file sans the key values of the template and function nodes containing source code values.
  • The flow.json file will contain filenames (in lieu of the source code values) that point to the raw text source code values in the newly created folder.
  • The filename of the source file will match the node name (iterated if need be) or the node type if no name was set i.e. function1.js, template2.html, etc.
  • The filenames will have extensions matching their content (ie function nodes create .js files, template nodes will honor the “Syntax Highlight” setting to determine the filename extension, ie .md, .py, .html, etc)
  • flow2src will accept an optional alternative folder name for the output; allowing for easy comparison of two folders should the original flow file change.
  • flow2src can have a “monitor” mode (exit-able via cntrl+c) that will watch the original flow file for changes and updates the folder output files.
  1. src2flow - a command line tool that reverses the flow2src; given a valid folder name, it will create/overwrite the flow file with the contents of the folder’s flow.json, replacing the aforementioned filenames with their escaped values. Thus, recreating the native Node-RED flow file format. Additional options could be:
  • JSON pretty print or compact the flow file.
  • A parameter to create the flow file as a different name.
  • A “monitor” mode that watches the folder for changes and re-integrates the flow file automatically.
1 Like
2

Additional thoughts:

  • support to target or exclude/include subflows containing function and template nodes.
  • Placing subflow function and template node source within a subfolder of the same name as the subflow.
3

This is a great list of requirements, Steve... and not just because it matches my thoughts almost exactly! ;*)

A year ago I started down this same path, implemented via node-red flows instead of scripts. My hope was to prove out the feasibility using flows before moving the solution into something like a storage plugin (so it does the translations with every deployment).

Of course, life gets in the way sometimes, and in this case I changed my career path to one that no longer allows me to use node-red on a daily basis. My personality on the other hand compels me to finish at least a feasible proof-of-concept. I guess what I'm saying is that I would love to share what I have done and help move a solution forward, if that is helpful.

Thanks for getting this conversation (re)started. I'll have to dig around to find the flows, and some samples of the source directories that were created through them...
__
Steve

1 Like
4

Ok, here is a sample of the output from my flow that "zips" up the flow file into a directory structure, suitable for committing to a source code repo...

image
image1727×761 108 KB

On the left you can see the individual "tab" folders, created to match the "tabs" across the top of the flow editor. Inside are .json files for each node -- and if any multi-line text fields are found (e.g. function node text), it is placed into a file named after the pattern <field>.<type> (e.g. the func.js file shown).

There are some unfortunate side-effects of trying to decompose a random JSON structure into a directory tree -- in order to ensure unique filenames, I chose to include the node.id value in the folder and file names. But I also tried to show the node name where it made sense, to make it easier to find the node you are looking to edit inside an IDE.

I really like the idea of using the "Syntax Highlighting" value to set the file extension... I guess that would be a simple enhancement. My intention was to have another flow reassemble the flows.json file from the directory structure, but I never got that far. Here is the flow that I have so far -- N.B. it does require a contrib node for creating the zip file:

image
image1575×641 93.3 KB

When you hit the http endpoint http://<host>:<port>/red/zip/flows it downloads a zip file that can be extracted into wherever you want to backup or commit your flow source tree.

flow source here 
[
    {
        "id": "2439e87451b07802",
        "type": "comment",
        "z": "5ec1131187731246",
        "name": "Extract the runtime flows.json file\\n into a zipped directory tree",
        "info": "",
        "x": 210,
        "y": 500,
        "wires": []
    },
    {
        "id": "25085272ac58faf4",
        "type": "http in",
        "z": "5ec1131187731246",
        "name": "",
        "url": "/zip/flows",
        "method": "get",
        "upload": false,
        "swaggerDoc": "",
        "x": 190,
        "y": 560,
        "wires": [
            [
                "6c9f89f19c8565ca"
            ]
        ]
    },
    {
        "id": "32ce4581186384c1",
        "type": "inject",
        "z": "5ec1131187731246",
        "name": "test zip flows",
        "props": [
            {
                "p": "topic",
                "v": "$now('[Y][M02][D02]')",
                "vt": "jsonata"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "x": 210,
        "y": 600,
        "wires": [
            [
                "6c9f89f19c8565ca"
            ]
        ]
    },
    {
        "id": "ec8b17d3cf1caf66",
        "type": "debug",
        "z": "5ec1131187731246",
        "name": "flows out",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "payload",
        "targetType": "msg",
        "statusVal": "",
        "statusType": "auto",
        "x": 940,
        "y": 560,
        "wires": []
    },
    {
        "id": "6c9f89f19c8565ca",
        "type": "http request",
        "z": "5ec1131187731246",
        "name": "GET /admin/flows",
        "method": "GET",
        "ret": "obj",
        "url": "http://localhost:1880/admin/flows",
        "tls": "",
        "x": 450,
        "y": 560,
        "wires": [
            [
                "ec8b17d3cf1caf66",
                "0e534a79b3b89185"
            ]
        ]
    },
    {
        "id": "92889fe06ee5d09a",
        "type": "switch",
        "z": "5ec1131187731246",
        "name": "msg.res?",
        "property": "res",
        "propertyType": "msg",
        "rules": [
            {
                "t": "istype",
                "v": "object",
                "vt": "object"
            },
            {
                "t": "else"
            }
        ],
        "checkall": "true",
        "repair": false,
        "outputs": 2,
        "x": 780,
        "y": 700,
        "wires": [
            [
                "d5f8028683a4a7d3"
            ],
            [
                "248320dfd65d34f6"
            ]
        ]
    },
    {
        "id": "b39272e50866b262",
        "type": "http response",
        "z": "5ec1131187731246",
        "name": "",
        "statusCode": "",
        "headers": {},
        "x": 930,
        "y": 840,
        "wires": []
    },
    {
        "id": "487ca486.c231ac",
        "type": "zip",
        "z": "5ec1131187731246",
        "name": "create zip file",
        "mode": "compress",
        "filename": "flows_shr.zip",
        "compressionlevel": "0",
        "outasstring": false,
        "x": 570,
        "y": 640,
        "wires": [
            [
                "92889fe06ee5d09a",
                "38824e8c756d1991"
            ]
        ]
    },
    {
        "id": "0e534a79b3b89185",
        "type": "change",
        "z": "5ec1131187731246",
        "name": "filename + filedata",
        "rules": [
            {
                "t": "set",
                "p": "filename",
                "pt": "msg",
                "to": "'~/backups/flow_dev.' & $now('[Y][M02][D02]') & '.zip'",
                "tot": "jsonata"
            },
            {
                "t": "set",
                "p": "headers.content-disposition",
                "pt": "msg",
                "to": "'attachment; filename=\"' & (filename ? filename : \"flows.zip\") & '\"'",
                "tot": "jsonata"
            },
            {
                "t": "set",
                "p": "payload",
                "pt": "msg",
                "to": "(\t    $tabnodes := payload[type='tab'];\t    $nodegrps := payload[type='group'].nodes@$n {\t        $n: {\t            \"id\": $.id,\t            \"name\": $.name\t        }\t    };\t\t    payload#$i[].(\t        $element := $;\t        $tabnode := (type = 'tab' ? $ : $tabnodes[id = $element.z]);\t        $grpnode := $lookup($nodegrps, $element.id);\t        $elename := $element.[label, name, summary, id][0].$replace(/\\\\n.*/, '');\t        $tabname := z ? 'tab.' & ($tabnode.label ? $tabnode.label.$replace(' ', '_').$replace(/[^\\w_]+/, '-') : z);\t        $grpname := $grpnode ? 'group.' & ($grpnode.name ? $grpnode.name.$replace(' ', '_').$replace(/[^\\w_]+/, '-') : $grpnode.id);\t        $typname := type.$replace(' ', '_') & '.' & ($elename ? $elename.$replace(' ', '_').$replace(/[^\\w_]+/, '-') : id);\t        $filepath := [$tabname, $grpname, $typname]~>$join('/');\t\t        [\t            /* output the entire node's json */\t            {\t                \"filename\": $filepath & '.json',\t                \"payload\": $string($element, true)\t            },\t            /* handle multi-line text content */\t            $spread($element)[$string($.*).$contains(/\\n/)].(\t                $fieldname := $keys($)[0];\t                $fieldext := $fieldname = 'func' ? 'js' : 'txt';\t                {\t                    \"filename\": $filepath & '/' & $fieldname & '.' & $fieldext,\t                    \"payload\": $.*\t                }\t            )\t        ]\t    )\t)",
                "tot": "jsonata"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 510,
        "y": 600,
        "wires": [
            [
                "487ca486.c231ac",
                "3641660dd8894cd5"
            ]
        ]
    },
    {
        "id": "38824e8c756d1991",
        "type": "debug",
        "z": "5ec1131187731246",
        "name": "zip output",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "payload",
        "targetType": "msg",
        "statusVal": "",
        "statusType": "auto",
        "x": 940,
        "y": 640,
        "wires": []
    },
    {
        "id": "3641660dd8894cd5",
        "type": "debug",
        "z": "5ec1131187731246",
        "name": "zip input",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 940,
        "y": 600,
        "wires": []
    },
    {
        "id": "248320dfd65d34f6",
        "type": "file",
        "z": "5ec1131187731246",
        "name": "",
        "filename": "filename",
        "filenameType": "msg",
        "appendNewline": false,
        "createDir": false,
        "overwriteFile": "true",
        "encoding": "none",
        "x": 940,
        "y": 720,
        "wires": [
            []
        ]
    },
    {
        "id": "152de0ddadb83a06",
        "type": "catch",
        "z": "5ec1131187731246",
        "name": "zip errors?",
        "scope": [
            "487ca486.c231ac"
        ],
        "uncaught": false,
        "x": 580,
        "y": 780,
        "wires": [
            [
                "a273e46259506636",
                "450dfb0216c54154"
            ]
        ]
    },
    {
        "id": "a273e46259506636",
        "type": "debug",
        "z": "5ec1131187731246",
        "name": "zip errors",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": true,
        "complete": "error",
        "targetType": "msg",
        "statusVal": "error.message",
        "statusType": "msg",
        "x": 940,
        "y": 780,
        "wires": []
    },
    {
        "id": "255e4104ae3dad7d",
        "type": "change",
        "z": "5ec1131187731246",
        "name": "",
        "rules": [
            {
                "t": "move",
                "p": "error",
                "pt": "msg",
                "to": "payload",
                "tot": "msg"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 760,
        "y": 840,
        "wires": [
            [
                "b39272e50866b262"
            ]
        ]
    },
    {
        "id": "d5f8028683a4a7d3",
        "type": "http response",
        "z": "5ec1131187731246",
        "name": "",
        "statusCode": "",
        "headers": {
            "content-type": "application/zip"
        },
        "x": 930,
        "y": 680,
        "wires": []
    },
    {
        "id": "450dfb0216c54154",
        "type": "switch",
        "z": "5ec1131187731246",
        "name": "msg.res?",
        "property": "res",
        "propertyType": "msg",
        "rules": [
            {
                "t": "istype",
                "v": "object",
                "vt": "object"
            }
        ],
        "checkall": "true",
        "repair": false,
        "outputs": 1,
        "x": 740,
        "y": 800,
        "wires": [
            [
                "255e4104ae3dad7d"
            ]
        ]
    }
]
1 Like
5

Nice! That's giving me some good insight. I started a prototype of the flow-to-source file generating command as a subflow in lieu of a CLI routine. But then this led me to the inability to discover the project path. Looks like the ability has been requested but it's still not possible from a subflow (discussion from 2019 at Project directory path - #3 by ianmacs).

So I had to implement it as a written node "from scratch" (thankfully, I have another set of subflows that automates that; but I have digressed...).

I've written the flows2src node (light green in the pic) that generates a "src" folder and then subfolders as the same name as the flows tab, and/or subflows (as subfolders). Within the subfolders are the template and function nodes as appropriately named files with extensions. I've implemented the filename iteration scheme to address two nodes of the same name.

Lastly, a manifest.json file exists in the root of the src folder that maps the files, object ids, and property that is being written (templates, functions for on message, on start, on stop). If the function property is empty it won't bother writing the file (git doesn't like empty files anyways).

I did have the node perform this after every. deploy but in the end decided to give it an input so that one can simply attach an injector node to do the same thing; or write out the src folder on demand.

I'll publish the node to npm and the flows directory shortly. Writing the reverse, given the manifest file, should be trivial.

flow2src
flow2src1024×597 74.8 KB

1 Like
6

If you want to make function code more easily diff-ed by git, for example, if you make sure flowFilePrettyPrint is set in settngs.js and then run this command on the flows file
sed -r 's/\\n/\n/g'
and send it to, for example, flows.json.formatted you will get a file with the function node split into multiple lines. I commit that as well as the flows file and can then diff the formatted files to see what has changed.

1 Like
7

That's a nice hack; however it still leaves us with one gigantic flow file and no opportunity for other IDEs. The initial flow2src node can be found here:

8

I was not suggesting it does everything you want, just throwing it in as for some (me at least) that, in combination with git, is sufficient. I don't have a gigantic flows file so that is not a problem.

1 Like
9

No worries Colin. All suggestions are good and I appreciate your feedback; and sed is super fast, and simple. I just wish I/someone could figure out a regex that re-integrates any changes quick n' easy. :slight_smile:

10

Version 1.0 of node-red-flow2src is here and includes the ability to generate a src folder (optionally automatically) for edits in your favorite editor, view git diffs, etc.; as well as the ability to re-integrate your edits back into your flow.

2 Likes