We use the "Projects" feature, and use GitHub to manage version control of our flows. The PrettyPrint setting has given us the ability to curate changes, but we also write a number of custom functions. Within a flows_*.json file, the contents of a function have the line feeds (\n) escaped. As a result Github can't discern changes line changes within a Node-Red function in a diff. If a Node-Red function node was changed, the entire "func:" field is marked as a change.
I'd like to propose that the setting for PrettyPrint be extended (if possible) to format the contents of the "func" keypair with linefeeds. With this change it would be possible to see an actual line change within a Node-Red Function node in GitHub, and manage a merge conflict within a Node-Red Function node.
Thanks for the time taken to consider my proposal.
I'm aware of this issue and would like to find a way to play more nicely with comparing changes to individual node properties.
The problem is JSON does not support line-breaks within a property value - so if we didn't escape line breaks, it would no longer be valid JSON.
There are extensions to JSON that support line breaks, such as https://json5.org/ - but by virtue of being non-standard extensions they reduce portability of the the flow file.
We know of some users who have implemented a YAML-based storage plugin to Node-RED to help address this - although this predates the Projects feature and you can't have a custom storage plugin and use the Projects feature.
But this is an area we want to look at. For example, allowing a flow to be split into multiple files. Or saving in an alternative format that does support linebreaks, such as YAML.
I agree that intentionally making the JSON unreadable, or non-standard isn't a good outcome. Understanding JSON is a big issue for people new to using the notation for Objects and Arrays. Adding the complexity of a non-standard feature would render JSON reading and edit services unusable. I admit this would add confusion rather than improve Node-Red.
I take your point on separate JS files. I saw this contrib node in NPM when I went searching for answers, https://www.npmjs.com/package/node-red-contrib-file-function from 2015. I haven't tried to use it, because I was concerned it wouldn't "play nice" with the Projects feature. This seems like the basic approach you described in your reply. However, the obvious downside is this would eliminate the ability to simply have JSON portability where you can copy and paste a flow. This is a pretty basic part of the Node Red culture that would be lost, or at least complicated.
I was thinking that this might be the answer. So instead of trying to alter the JSON in flows_*.json we work with it, and replace {"func" : "code"}with {"func" : ["line of code 1", "line of code 2", "line of code 3"]}.
Then the existing PrettyPrint would render it as:
In this way Node-Red could ensure compliance with the existing JSON standards and add the capability to version control a function. Generally all "func" properties could be put inside an array, even if there was only one line of code.
I assume this might impact the core processor a bit for Function nodes, but it seems like it would fit in with the rest of the existing Node-Red code base.
As Nick said (recently, and last year), the problem is with breaking the JSON formatting, which does not allow for either line breaks or comments :*(
One solution that is currently available is to save your flows as YAML (a recursive acronym for "YAML Aint Markup Language", oddly enough). In practice it is a superset of JSON without all the quotes/braces/brackets. So it's a bit easier to type, but requires you be diligent about indentation. Check out Nat's contributed code that let's you replace the usual JSON formatted flow files with a YAML version. He has some nice comparisons between the two versions in the documentation. And the process to switch over is pretty painless -- modify the settings.js file to reference the installed module, and the next time you deploy your flow it will be saved in the new format. Of course, the extension is .yaml so your Git project will see the two .json files as deleted, not modified, but hey...
If you enable a custom storage plugin, the entire projects feature gets disabled. This is because almost all of the projects logic lives within the default local filesystem storage plugin.
Doesn't stop you managing your flow files in git outside of the editor, but you won't get the integrated version control within the editor.
Oh, that's right -- I forgot about that part... I guess you can tell that I'm not using projects ;*)
Did I see a mention at one point about embedding Nat's conversion logic into the filesystem storage plugin? It would be nice if that conversion step could simply be triggered by the flow filename's extension.
But this is an area we want to look at. For example, allowing a flow to be split into multiple files. Or saving in an alternative format that does support linebreaks, such as YAML.
Could you comment on my idea (above) for adding an array of strings to the "func" key/value. I was thinking this could keep the JSON standard while adding the necessary LFs to do a GitHub diff on a Function node. Seems like this would be a fix that can extend Node-Red's capabilities without impacting the core operations.
Nick is somewhat busy on 0.19 see https://discourse.nodered.org/t/0-19-node-red-roadmap/
While it may be a valid idea it realistically won’t get serious consideration until after 0.19 is released and initial bugs squashed.
What are the performance implications of having to demand hall arrays vs using yaml or multiple files etc.
But is JSON the right format for the future? If we have to come up with workarounds simply to make everything fit, wouldn't this be a good time to reassess - obviously while keeping a backward compatible option.
There is a lot to think about here which is why Dave is encouraging us to give Nick a break!
Personally, I think that I'd like to see a future option that allows flows to be kept in separate files with functions in native JS files. Folders could also be used to separate out flows (tabs) if desired and would certainly work to separate out projects. Still lots of options to decide on though and a fair bit of performance testing would be required.
But, of course, this is open source, so you are always free to experiment and we (the community) love to see ideas.
I was actually trying to do something simple, diff a function node in Git/Github/VSCode, and be able to see and edit the changes more easily.
I'd like to see separate JS files for function nodes so that you could edit them in VSCode or another IDE, but there seems to be a number of implications.
There would need to be some kind of compile step to turn it all back into JSON to maintain portability. HyperLedger Composer does something similar with chaincode when you create a business network archive (BNA) file. The archive makes the business network portable.
The challenge I see with this approach is that you'd have a JSON file and JS file that may have changed, and a potential merge conflict in two places for one change in Github. When you do this in HyperLedger you just gitignore the BNA files, and rebuild them from the source.
I run the following command on my flows file before committing each time
sed -r 's/\\n/\n/g' <flowfile>.json <flowfile>.json.formatted
This generates an extra file with the functions reformatted and a diff on this shows changes in a reasonably clear way. If I was clever enough I expect I could make git do this automatically.
From what I've read, you can put that command into an executable file called pre-commit under your <project>/.git/hooks directory -- something like this (untested):
#!/bin/sh
sed -r 's/\\n/\n/g' <flowfile>.json <flowfile>.json.formatted && git add <flowfile>.json.formatted
Most of the sample scripts have lots of additional error checking, which would be good, too
Thanks @Colin & @shrickus. I did consider this solution, but I think the problem is that it doesn't really solve the Git merge conflict problem.
The problem I've encountered with a large Node-Red project with multiple Git contributors is that you'll naturally get into situations where you end up with a merge conflict, and then when you look at the diff of the flow file (unformatted), you are going to have to edit the functions ("func":"<function as a string>") that are collapsed into a single string.
Maybe there is a way to modify PrettyPrint with an argument to decode "func": "<string>" in a way that would meet the need, and format the "\n" inserted by the editor into the flow.json.
If we create a formatted flow file, then we will have two files that have a merge conflict.
Back to my original idea, I'd like to come up with something that gets us past the original comment from @knolleary where we need to produce a valid JSON object for the value of "func":,and it can't be LFs in JSON. My proposal here is that we use an array of strings, one array location for each line of the script as the value for "func":.
I noticed today that someone forked the file-function node in the last 4 months and updated to work with the previous version of Node-Red. This provides a vehicle for a possible solution.
I was thinking about looking to add on this additional formatting of the function script to that process.
The question for @knolleary and the team is if a new function-like node could deliver the key/value"func":"<string>" as "func":[<string1>, <string2>, ... <stringN> would the runtime understand how to use the value or would we need to build a new parser and other parts. I'm a novice at what's inside the core handling this script loading process.
If this all could work, then we would end up with a solution that has the script saved in .js files in a git-ignored directory, and a flow.json that has each line of a function-like node formatted separately for ease-of-use when deciphering a git merge conflict. In a conflict you could find the change in the Diff window, and make the change in the .js file.
It's an old post, but it is still relevant for us.
I love Node-RED but a big drawback is the cooperative work. It is hard to merge parts together. There's always conflicts that would not occur in a typical way of programming.
The way I see it, flow file should be splitted in many parts. It should never gros up over 10k of lines. So maybe YAML could be a good idea. But, I think JSON if fine as long a the code is splitted in smaller pieces.
So here's what I suggest:
Each flow should be a .json file
Each subflow should be a .json file
Each function node should store his content into .js file
Each template node should store his content into .txt file (or any other appropriate file)
Each template (Dashboard) node should store his content into .html file
Even Jsonata expression should also be in separate file
Maybe I'm going too far, but it may worth considering to split each nodes in separate file and to regroup them into folder representing flow/subflow
Each file would be name with a prefix and node id:
flow_f6172214.8c591.json
function_"92c036c9.c74c08.js
template_2a442021.5e674.txt
template_ui_a0bb0095.d3bed.html
jsonata_dfa90ba3.755df8.txt
Then
There would be a main.json refering to each flow file
Then in each node, there would be a reference to files it need
Benefits
There would be less conflicts in git as each added nodes would be splitted to manye files
When resolving conflicts, it would be easier to see which flow each node belongs to.
It be easier to push/publish only parts of a project.
