Best practice: custom v. function node

[gregorius]

@Colin

1% would be able to develop a new custom node.

There are millions of Linux users but only 0.01% of them write device drivers yet there are still books on writing linux device drivers.

In this case absolute numbers are more interesting, 0.01% of Linux users is still thousands (tens of 1000s?) of people. 1% of Node-RED users is about ~2700 looking at the numbers. There have been about ~2700 unique package maintainers and ~700 have been active in the last 12 months. The trend of these numbers is upwards.

With an upward trend it might well be a good idea to have guidelines for two reasons: 1) to provide a clear starting point for new folks and 2) to keep a unified UI within Node-RED. Apple is famous for its design guidelines and Apple provides tools to make it simple to implement those guidelines - hence every interface of an Apple device is similar and similarly approachable.

In Node-RED, an example of unified UI is including the name field when creating a new node. Every palette node has a name field. That's a UI design guideline, an unwritten rule. What is not defined is when does the name field go at the top of the config panel (see function node) or at the bottom (see split, join, batch and sort nodes)? A unified UI makes approachability for new folks much simpler.

(If you believe the location of the name field is a laughable example, try moving the quit menu point to somewhere else when developing an Apple application ... it just won't happen not because its Apple being pedantic, its because Apple users expect the quit menu point to be in the same place for all applications. Similarly I expect the name field for a palette node but I expect to be at the top of the config panel.)

[gregorius]

@dceejay

The reason we don’t want to allow colour change is because of readability of the flow.

Right, I'll stop working on that PR ...

If I can then ask, why is that different for subflows? After all the functionality included in a function node can be more than a subflow, so the impact on functionality of both is similar. Or even better, I can include my function node in a subflow and then change the colour of the subflow.

I do suffer from being a pedantic person, I mean no malice.

[Trying_to_learn]

@gregorius

My phrasing was too dogmatic, of course freedom should remain but for people new to Node-RED, it would be nice to have guidelines when to use what when extending Node-RED. So yes, you are right there should not be a must but more along the lines use a custom node if you want XYZ if you don't want XYZ then use whatever - my example for XYZ are certain ESM packages but even that is a grey zone! Ah, I have one: if you wanted a different coloured node, then you'll need a custom node (damn, you could also use a subflow)

No worries.

I know sometimes words fail to convey the real meaning.

I also was kind of playing devil's advocate there too.

Glad you are enjoying using NR.

I honestly don't believe how far / much I have done with it in such a short time.

[gregorius]

@Trying_to_learn

I honestly don't believe how far / much I have done with it in such a short time.

Have the exact same feeling. I wonder for myself why I have this feeling. I think its partly because the development process is so much "smoother" in Node-RED - it seems quicker but in fact, I would have achieved just as much using a text editor? (I've used Flask with automatic reload and RoR with automatic reload - i.e. code change caused the application server to reload all dependencies.)

To put it another way, using Node-RED there always seems something happening (dragging debug here, straightening a line there, ...) - this gives me the feeling that it a continuous process without interruption.

[Trying_to_learn]

For me, that is only sometimes.

A lot of the time it is seat of my pants code writing.

Good/Bad. But it is what it is.

[gregorius]

@Trying_to_learn

For me, that is only sometimes.

Ok full disclosure: I have 6 instances of Node-RED open in 6 tabs in my browser! Each instance has its own purpose - two raspberries for node development, two instance on heroku for my web apps and two locally installed instance for mind-map and trying out ideas. I'm continually copy & pasting nodes across instances - basically its a staging, development and production server setup in the browser.

[Trying_to_learn]

I have.... 3 RasPies running all the time. All headless.

Each has.... > 10 tabs running. (Nominal)

And it is getting more complicated by the month. I seem to love making my life miserable.
But sorry, I don't want this to turn into a pissing contest.

If I think back.... 5 years (there about) I knew NOTHING about Node-Red.
I'm not sure I now know more or not, but gee, I've done some amazing things with it.

And the newer releases! WOW! Some of the benefits are just out of this world.
I am now a subflow addict - using them all over the place.

[ralphwetzel]

@gregorius

If I can then ask, why is that different for subflows?

That was a design decision!
As such it leads us directly into one of the core topics of this exchange: The difference between an Apple driven product vs. Node-RED stems (my perspective) from the fact that the latter prospers from the non-refunded contributions of a community. A contributor - usually being among those 1% = quite skilled users - might have a different view on those 'guidelines' & perceive them as e.g. too restrictive or simply not-applicable (for this, his special, case ). And we all know the story: The effort to change a guideline - even with good reasoning - is tremendous. As such, people accept those, the guidelines fix a product at a given state & the product deteriorates. Apple counteracts this with a powerful design team - and always had a strong focus to ensure to set trends.
As there's no design team for NR (and this is not a call to put one in place!) I prefer to live in & contribute to a vibrating environment of ideas and constant challenge of the status quo. For a great example of this "in action" I refer to the exchange we had recently around "the editor is no dashboard". That was true for years (despite not always complied with to 100%) - until someone had a proposal that got a lot of attraction. I'm quite convinced, we'll see an evolution of Node-RED around that topic in the coming months.
Does a "guideline" prevent such a story from happening. Not at all - yet our experience tells us, it's going to be far more difficult, once "guidelines" take the lead - and thus creating far less good ideas! People need a lot of courage to question guidelines - knowing there's one argument they can never beat: "This is the guideline..."

[dceejay]

@gregorius

If I can then ask, why is that different for subflows?

The reason we "allowed" sub flows to have colour option was because we wanted them to be exportable as "nodes" - Packaging a Subflow as a module : Node-RED - as a way of creating nodes out of sub flows in a low code manner, and if someone creates a node they need to be able to set it's icon and colour...

And re position of name field... Originally we had it as default as the last field on every config page, mainly as it was optional and consistent (as you like . But as nodes got more complex, (ie config got longer and may need scrolling) - and others made more sense with it at top - eg functions where you really ought to name them... then it has migrated to be at the top.... But yes - in general all new nodes should have the name field at the top to encourage folk to name their nodes.. and indeed it is one of the things on (one of) my to-do lists to revisit the core nodes and audit and update them for v4.

Likewise - once upon a time - we tried to make config panels read like a sentence... Eg TCP-in - Listen on port xyzzy (and) output a stream of strings delimited by \n. etc - rather than have a bunch of random options. But again that may not translate to other languages and some things are too complex... but... hey..

[gregorius]

@ralphwetzel

Does a "guideline" prevent such a story from happening. Not at all - yet our experience tells us, it's going to be far more difficult, once "guidelines" take the lead - and thus creating far less good ideas! People need a lot of courage to question guidelines - knowing there's one argument they can never beat: "This is the guideline..."

But was not the notion "the editor isn't a dashboard" a definite and to-be-taken-seriously guideline? A well documented and set-in-stone guideline? Then someone came along and took another approach, even though there was the guideline? And now that guideline is being questioned. Welcome to evolution

Guidelines are just that: a guide. What is a guide? A guide is someone who knows the terrain and shows you a - possible - path through that terrain. Its still left up to you to follow the advise of the guide. But for those who are uncomfortable within the terrain having a guide to take them along the path can be comforting.

I don't think that a guideline makes a difference, it's the enforcement of the guideline - which is done by humans - that needs to softening sometimes (perhaps).

Design patterns, best practices, design guides, knowledge bases, guidelines &c are all the same effort of transferring wisdom to those starting out. As we were all young, we know that most of the time that wisdom isn't absorbed. Strangely not because the young aren't willing to learning, mostly because the wisdom is packed in such a way that experience is required to understand the wisdom. Experience that the young have yet to make. (sorry that was completely off-topic!)

[gregorius]

@dceejay

and indeed it os one of things on (one of) my to-do lists to revisit the core nodes and audit and update them for v4.

Again this is the pedantic me, not the malice me but:

From the perspective of a Node-RED user, this would be the most important feature. Consistency in the UI makes a major difference to all users and anything that makes the UI better/consist/smoother/clearer/understandable should be priority 1. It should be at the top of your todo list! Get it done!1!!! /s

From the perspective of a developer who enjoys hacking code, UI is the worst most boring thing. That's why Apple builds tools for developers to use to create UIs.

(This comment should be seen as a generalisation and not specific to individual developers.)

[dceejay]

Most of my to-do lists are non-Node-RED ones sadly - some imposed by financial necessity and some by spousal demand (but yes I hear you)

[dceejay]

@gregorius

"the editor isn't a dashboard" a definite and to-be-taken-seriously guideline?

YES ! - We didn't want to get to the point where users need to expect to have a display open all the time to run Node-RED. It is meant to "just" be a dev tool... do you run apps in VScode, or Sublime, or another editor ? Also as soon as it becomes a dashboard then "everyone" has an opinion on layout, fonts, colours, etc and we would spend more time chasing down those paths than developing the core execution capability - hence we have tried to be fairly firm on the idea that the editor is for creating flows, (and indeed has a set of admin privileges), and then the display of data is a second layer (with user privileges) that people with far greater UI skills can customise however they like to suit their particular use case.

BUT ! - of course we recognise that for debug there may be better ways to help and so there are some soft edges to this and some indicators, images, etc have been added, and no doubt more will be. But as you say - evolution will happen.

[gregorius]

@dceejay

It is meant to "just" be a dev tool...

Node-RED is an operating system, not a dev tool!

If one takes that perspective, one gets to interesting ideas: Node-RED should be made extendable but not extended itself. I.e., allow nodes and plugins to get deeper into the Node-RED workings. Node-RED should allow users to define what happens in the info box via plugins (for example).

Also if its an OS, then UI guidelines are important since you don't want to end up with a GUI nightmare (even though its much better now) that Linux is/was/had. This is already partly addressed through the collection of widgets which already provides for UI unity.

But in all seriousness, NoROS (Node-RED OS) is the first visual flow based operating system. From that perspective Node-RED should be doing a lot less and allow plugins and &c to do more.

A thought experiment: what if Node-RED were to become an OS, what would happen? It has all the makings for one: module based architecture, clear APIs for extendability, a solid base functionality (it takes good care of executing flows).

The question remains: can Node-RED be developed in Node-RED? (After all it's a dev tool and operating system )

[TotallyInformation]

@gregorius

Ok full disclosure: I have 6 instances of Node-RED open in 6 tabs in my browser! Each instance has its own purpose - two raspberries for node development, two instance on heroku for my web apps and two locally installed instance for mind-map and trying out ideas. I'm continually copy & pasting nodes across instances - basically its a staging, development and production server setup in the browser.

That sounds ripe for some automation! A quick flow to promote a flow from dev to UAT to live or whatever environments you are using. No more copy/paste!

@Trying_to_learn

I have.... 3 RasPies running all the time. All headless.

It was when I got to 3 Pi's that I decided to repurpose an old laptop and consolidate down to 1 device.

@ralphwetzel

The difference between an Apple driven product vs. Node-RED stems (my perspective) from the fact that the latter prospers from the non-refunded contributions of a community.

Well said. Thinking back to the arguments I've had with senior Apple folk (benefit of working for a national NHS body) about their approach to the world ... Their approach works well for a common sub-set of people but it isn't flexible.

Node-RED has managed to walk a line of both ease-of-use AND flexibility and that is no mean feat.

@gregorius

But was not the notion "the editor isn't a dashboard" a definite and to-be-taken-seriously guideline?

Originally most of us did take it seriously and people were strongly discouraged. Only in recent times have we come to a softened stance I think. And this is a good approach. Keep things tight at the start because once you open the door, there is no going back. So don't open the door until you understand the consequences of the change.

@gregorius

Design patterns, best practices, design guides, knowledge bases, guidelines &c are all the same effort of transferring wisdom to those starting out.

I don't think any of us are arguing against these things. We would all love to see more and better documentation, design guidelines and knowledge sharing. But someone has to make it all happen and it needs to be at least somewhat coherent. I know that the core team have tried to ensure that Node-RED documentation has a common feel to it to try and keep it consistent. But we all know that there is lots that isn't documented fully - but there is only limited resource available. And adding to the official documentation is actually quite a commitment for anyone not already versed in GitHub based development - in terms of having to discuss a potential change in the forum, set up an environment, clone the repo, do the edits and submit a PR. If you don't do these things every day, it can be quite time consuming and is a barrier to engagement.

@gregorius

From the perspective of a developer who enjoys hacking code, UI is the worst most boring thing. That's why Apple builds tools for developers to use to create UIs.

And why I'm looking at ideas to make this easier for Node-RED node developers

@gregorius

The question remains: can Node-RED be developed in Node-RED? (After all it's a dev tool and operating system )

Not quite convinced about it being an OS. There are absolutely things I would never recommend doing in Node-RED, certainly in a production or commercial environment. Just because you have a hammer, not everything is a nail.

Node-RED flows and the environment carries a lot of "baggage" due to its nature. This is not a criticism, merely an observation. And all of its overheads operating in the context of Node.js which, while thanks to the ongoing efforts in V8 and elsewhere is amazingly performant, it is still an interpreted language (mostly). So eventually you start to hit limits of performance and resource utilisation. You also hit restrictions due to 3rd-party libraries such as ExpressJS and Socket.IO which themselves are complex. Again, this isn't a criticism, merely a note of caution about not making every problem into a Node-RED shaped problem. We don't server the community well if we do that, nor do we do justice to Node-RED as a platform (which is what it is, not an OS).

[Trying_to_learn]

@TotallyInformation

It was when I got to 3 Pi's that I decided to repurpose an old laptop and consolidate down to 1 device.

Yes, but then you don't/may not know if the laptop dies.
This way I have all 3 of them checking one another and if one dies, the other two can tell me.
If two die, the remaining one can tell me.

Yeah, if all 3 die.... But that's a whole other scenario.
It is kind of a redundancy having 3 devices rather than one.
I don't think it will get any bigger now though.

Just other devices with other functions.
(But sorry, I digress from the thread's question.)

Oops, I just realised: There are 4 raspies. New one just came online a week or so back.
But that is it..... He says

[gregorius]

@TotallyInformation

discuss a potential change in the forum, set up an environment, clone the repo, do the edits and submit a PR.

Hehe you forgot: and sign an NDA! I made a one byte change and had to did sign my life away

@TotallyInformation

Not quite convinced about it being an OS. There are absolutely things I would never recommend doing in Node-RED, certainly in a production or commercial environment. Just because you have a hammer, not everything is a nail.

From the perspective of a node, Node-RED is the operating system. It's just a thought experiment but one that from a nodes perspective is all too true.

The underlying point is of course that sometimes changing ones perspective helps to find other approaches. One can draw parallels and consider them. It can help to make decisions about which features are relevant and which aren't. I was thinking about the comparison and realised that Node-RED also has package manager and it even has device drivers (the nodes) since the API (i.e. message object) is clear and understandable and only provides the communication basis. Linux device driver API has a very limited number of core functions (if you look the file system APIs, then its even less) - this idea can be thought of as a guiding principle when it comes to extending Node-REDs message passing infrastructure - less is more. Again: I'm not advocating anyone do this, it's just me and my worldview.

@TotallyInformation

So eventually you start to hit limits of performance and resource utilisation.

All too true but before we get into the weeds, let us have a look from above. And again, I'm not advocating that Node-RED should become an OS, I'm encouraging folks to have look from above and see the parallels to other concepts. After all, Linux seems to be doing great with their concepts so perhaps there are some fundamental truths and learnings that can be taken onboard.

Btw using metaphors and analogies can help to explain concepts in terms that others can understand. If I went to a Linux developer and told them that Node-REDs concepts are similar to device drivers, they would understand that immediately. If went to a product owner and told them the same thing, they would wonder what a device is (in that context). So instead I could use the UML flowchart analogy and say that Node-RED flows are just UML flowcharts that actually get executed.

@TotallyInformation

That sounds ripe for some automation! A quick flow to promote a flow from dev to UAT to live or whatever environments you are using. No more copy/paste!

That's why I created flowhub.org - push and pull between Node-RED instance. Also those 6 instance are all different and only share some fundamental functionality, they are definitely not the same. Else I would use the SendFlow node to deploy to another instance

[gregorius]

@Trying_to_learn

Yes, but then you don't/may not know if the laptop dies.

What's the backup principle for Node-RED? There isn't one AFAIK. Hence I backup up everything I do to GitHub every 15 minutes (at least for the production installations). I created a flow that pushes the flow.json to a private repo and done. Of course only committing to GH if there was a change else I would have (24 * 60 / 15) empty commits per day

@Trying_to_learn

(But sorry, I digress from the thread's question.)

It's just a guideline that threads stay on topic (or at least I take it like that)

[Trying_to_learn]

I think confusion is creeping in.

The 3 pies are running and monitor things.

1 - A main one that overlooks everything.
2 - One which displays the state of things.
3 - My DNS and other things.

You also have to understand that this started with just ONE machine but it was so underpowered.... A RasPi with only 24 GPIO pins, and a yellow composite video output.
So it couldn't handle doing ALL the stuff.

The second one was a RasPi zero and it too got overwhelmed with what I was doing.

NOW they are all RasPi 3's.

Each also monitor one another and if one dies the others will notice and I will be informed.

As to backing up.... That's done now and then too. But isn't in the scope of what the machines do.

Though I have written a script that backs up the flows/etc to either a USB drive or copies in another directory on the SD card which I now and then copy off and store elsewhere too.