Join the json code from multiple flows into one file

General
1

I have a device with 8 channels I am controlling from Node-Red. I have setup 8 seperate flows called Ch1, Ch2, Ch3...Ch8. In those flows I named all my flow variables somethingCh1, somethingCh2, etc. I manually created the groups to contain the UI nodes from each flow and called those "groupCh1", "groupCh2", etc. I then got the id for each of those groups.

Now I can work on the flow for Ch1 and save it to a file on my laptop and I have a python script go through and find/replace anything "Ch1" and change it to "ChX" and change the corresponding group Ids.

Maybe there is a much better way, I am not sure, but it's pretty quick except for the next part when I have to go and delete all the flows and paste in the code from each flow 7 times (Ch1 is already in there so I have to delete Ch2-8 and paste Ch2-8). My knowledge of working with json is extremely limited, but it must be possible to combine all of these into one file that I can just paste because Node-Red already has the option to export "All Flows" as one file. Also, if I didn't have to manually create the UI groups that would be great, but there are only 8 of them and I only have to do this one time, so no big deal...until I have a device with 100 channels...

On a simpler Node-Red with just two flows, I tried a couple things with Python, but just ended up with an unusable file. Here is my attempt:

import json
import glob

result =
for f in glob.glob("C://Users//Desktop//python_json_practice//*.txt"):
with open(f, "rb") as infile:
result.append(json.load(infile))

with open("C://Users//Desktop//python_json_practice//merged_file.json", "w") as outfile:
json.dump(result, outfile)

When I try to paste this file I get: Error: Input not a valid flow - item 0 missing 'id' property

I really had no expectation the python script would actually work, so if anyone has any hints that would be great! Here is a partial output from my python script. The double brackets is probably the first hint something didn't go right.

[[{"id": "3cd6636b.34b4cc", "type": "tab", "label": "Flow 1", "disabled": false, "info": ""},

2

If you restructured your flow to not use flow context it might be much easier. What are you using them for?

3

In general I am using them as a way to store user entered setpoints and then update those setpoints when the device needs to modify them.

So for example a user enters a desired setpoint into a text box and I store that as flow.set("tempCh1"). Sometime later the device itself updates this temperature so that another node elsewhere in the flow can make use of the new temperature.

Or maybe a better example: when I set the device to run, I store a variable flow.set("sourceCh1", true). I have several functions in my flow that will run logic and make decisions based on whether "sourceCh1" is true or false.

It all works beautifully, except especially now in early stages of development where I make small tweaks to the flow quite often, it is painful to do all the copying/pasting.

I'm closer maybe on the python as I am using a dictionary for json.dumps, but there is some weirdness with quotations and I get an output that has obj1 and obj2.:

result =
for f in glob.glob("C://Users//Desktop//python_json_practice//*.json"):
with open(f, "r") as infile:
result.append(infile.read())

data = {'obj1' : result[0], 'obj2' : result[1]}

with open("C://Users//Desktop//python_json_practice//merged_file.json", "w") as outfile:
json.dump(data, outfile)

with open("C://Users//Desktop//python_json_practice//merged_file.json", "r") as infile:
clean_json = infile.read().replace('\"',""")

with open("C://Users//Desktop//python_json_practice//merged_file.json", "w") as text_file:
text_file.write(clean_json)

4

Now I can work on the flow for Ch1 and save it to a file on my laptop and I have a python script go through and find/replace anything "Ch1" and change it to "ChX" and change the corresponding group Ids.

This makes me wanna cry...

Maybe think about that the channel number is a variable inside of your program?

1 Like
5

There is quite possibly another way but I'm not seeing it. Another complication is that I really need all 8 channels to be on the same tab. Multiple users might come to the node-red so if they all started switching tabs it would change the tab on everyone's browser and that would be a problem :frowning:

6

This really sounds like a job for subflows

7

But all those "channels" do the same thing except for their name ?

8

actually I was going to try that, but until 0.20 came out I really needed the inherit from parent feature. Even with that, I'm not 100% sure that will work for me

9

I may not be seeing deeply enough into what you're doing, but couldn't you use a switch node to sort by channel and route the messages to identical instances of a subflow for processing?

10

I'll do my best to create an analogy:

Let's say I want to control 8 light bulbs and when I turn them on with a button I want the hypothetical variac controlling them to ramp from a user specified minimum voltage to maximum voltage at a user specified rate. I have one sensor that is able to detect the amount of light from each of the eight lightbulbs and outputs that also to the UI and I don't want it to surpass a user specified value. So let's say the user wants 1000 lumens but his voltage setpoint would eventually hit 2000 lumens, well node-red has to tell the variac controlling the lightbulb to stop increasing voltage. I also have values for how long I want the bulb to remain at it's maximum voltage before ramping down to the minimum voltage. I also need to allow the user to change all these setpoints at any time and start or stop the ramp any time and view the voltage and light output from each bulb. There is feedback required from the light sensor to the variacs controlling the lightbulb, so node-red will have to send commands to the variac to set its limits, and that command will look something like : "variac #x, limit: low, high". I also require all info for all 8 bulbs be observed on one tab, not multiple tabs.

I'm not sure this is even making much sense so maybe I'll stop there and see what the group thinks...

Based on everyone's feedback my logic definitely requires a look at how I am doing this, but I am really convinced I need some way to store variables, so I'm not sure why storing them as flow variables seems so crazy? Pretty much as soon as I started writing the flow I was thinking, "wait a minute, I have to store values somehow, how do I do that?" Then I discovered flow variables...

11

This actually makes quite a lot of sense. Since you're using JSON in your initial example, I'm assuming the device you're interfacing with accepts and outputs that as its standard form of communication. I'd recommend injesting the JSON and converting it immediately into a javascript object for easier manipulation. Your "CH1, CH2, etc." seem to be perfect "topic" properties to use switch nodes to send the message into the apropriate subflow instance. You can store your data as flow variables in the subflow without needing to worry about channel names in your code. If you have other variables that you do want to differ per subflow, you can pass a message to that topic to set the apropriate variable to be evaluated within the subflow.

I'm not even entirely certain files would need to be used at all here unless that is how your device talks to you.

I can bang something together when I get home, though the biggest unknown here is what your integrated device needs to see in order to do its thing.

12

Thanks for the feedback. We might be talking slightly differently though. The JSON in my original question is just my flow that I export. I'm not working with JSON in anyway at all in my flow. I have a python script that takes my "Channel 1" flow and finds/replaces all "Ch1" variables and changes them to "ChX" and writes a new text file. Then I paste that text file into Node-red so I have 8 flows "Channel 1", "Channel 2", etc. I just want to "combine" the 8 text files into one file that I can paste into NodeRed.

For example, I can use NodeRed to export "all flows" and then paste that right back in without issue. So if I can do that in Python with the 8 individual files I create, it would save me from pasting in 8 different flows everytime I make changes. It's not even a huge deal because right now I work on Channel 1 until I am happy but leave my Channel 2-8 running as-is. Then eventually I update all channels to the latest code. But if I could more easily copy in my flows I would more frequently just update all the changes.

Subflows might work for me because a few weeks before nod-red 0.20 came out I was working with them but discovered the inherit from parent feature was not ready. Even so, I'm thinking there is a big advantage to the way I'm doing this: I can work on my "Channel 1" flow and change all sorts of things and then simply run it through my python script to make exact copies for Channel2-8.

As for my device, it generally requires exec commands like: "set Channel 'x', minChx, maxChx" where 'x' is 1-8 and min and max are user specified (or node-red generated as a result from feedback from the device). So it is just a string that I run in the exec node.

13

Do you use MQTT? If not there are a number of advantages to using that for storing state such as setpoints and whether something is running or not. If you chose the topic names accordingly you could make all your flows the same except for one component of the topic, which could bring the differences down to just a couple of nodes.
Another advantage over context variables is that when a value is changed by one part of the flow any other parts that need to know (to display on a dashboard or whatever, automatically get informed.
Also on startup the value automatically get injected into any nodes that need to know.

14

Oh... so you're trying to dynamically update the node-RED flows from a separate interface entirely...

Let's try looking at it from the ground up. What input does the device you are controlling take?

15

I believe that is accurate. My device updates flows based on values it is sensing. i.e. it senses the light output from the light bulb and reports that back to node-red which updates a plot for example. My device takes inputs from the linux (raspberry pi) terminal.

16

I use MQTT for one aspect of the flow, but it is not critical to device control. My concern of using MQTT is that the variables I am storing are critical to device control. So if MQTT went down for some reason I could be in a pickle. Additionally I still have the issue where all my flows are the same but if I make changes to one flow I still need to update all of the flows to be identical which means I need to copy/paste or change names somewhere along the way.

No one has replied yet with any suggestions to merge my flows into one JSON object for easy import back into Node-Red after I modify all my variables. It has to be possible because Node-Red itself allows you output all your flows as one JSON object. I just don't know much about JSON or how to merge/combine JSON objects into one object.

Thanks

17

Perhaps you should post 1 flow so that we can see/help how to improve.

18

Why do you think mqtt might go down? I don't think I have ever seen mosquitto fail. For critical stuff you run it on the same machine as node-red of course.

19

Probably because that's not the best way to achieve this.

@JayDickson already suggested using subflows.

@bakman2 has suggested you post your flow (remember to post between 3 ticks or press the code </> button) so we can help you approach this in a more programmatic way.

I suspect you could break the common logic out and put it in a subflow then simply pass in the channel name and whatever else is required into 1 subflow. Massively simplifying things. No export / import necessary.

Subflows

20

With help of a buddy I figured out combining the flows is really no big deal. Each flow is a JSON object. So I have 8 flows and using Python I can just strip the bracket off the end of one and add a comma and new line and then strip the bracket off the front of the next one and add it to the first one and so on. Then I end up with one one JSON object that represents all my flows and I can paste that into Node-Red with one simple paste action.

So my total workflow is 1) copy my Channel1 to a text file. 2) Have python make 7 new flows by search/replace Ch1 for ChX and then merge all of those into one JSON object file. 3) Delete one by one the old flows (admittedly a little annoying but I don't think there is a way to delete multiple flows at once). 4) Paste my JSON object. It's pretty simple, probably less than 2 minutes and I only really have to do it once I'm fully happy with Channel 1.

I can continue thinking about suggestions some have made regarding subflows, etc. but I do not think it is possible or maybe even desired to do it that way. By working on Channel 1 and then making exact copies for Channel 2-8 I ensure all my changes are identical and I did not miss anything.