I would like to share something that I have been working on: RedBack - Node-Red as Backend. The point of the project is to utilise Node-Red as API endpoint for a python/jQuery frontend on top of a Postgres database.
Basically I had an idea of a bidding/auction platform that I wanted to build (as MVP) and thought I'd try Node-Red out as possible backend. Turns out that Node-Red (for me) was superb and I ended up being much faster than had I coded this using an editor-based programming language.
The flows.json is some 300k and I have no idea how this would scale (even though I have a few ideas). It would be great to hear from others who might have attempted the same! I spent a lot of time rearranging and adjusting flows. I make a lot of use of the link-in and link-out nodes. For me it's all very understandable and I can navigate the flows fairly easily, I wonder if this is the case for others?
Perhaps I am the only one who can see a real potential in using Node-Red as a backend but someone has to be the first! I spent many years building and talking about backends and many developers seem to enjoy the tapping of keys rather than the clicking of mice!
I hope I can inspire someone with this and thank you for Node-Red, it certainly inspired me
I guess this is a descriptive issue but I'm confused (doesn't take much!) - Node-RED is already a back-end? Do I assume then that this is a pre-wrapped app with node-red, Postgres, REDIS, Flask and Python built in?
The other thing that slightly confuses me - why Python/Flask for the front-end when Node-RED already has several tools for this? Is it just because that is what you are familiar with? Feels like too many technologies.
And apologies, not trying to put down your work - just trying to get my own perspective straight.
Sorry I should have this made it more clear. It seems to me that NR has a specific use case and that is IoT and dashboards to display current situations of IoT devices. It is not thought of as a possible API backend to access data or logic engine to implement business logic. Again this is my impression.
It has a fabulous way of representing logic, specifically business logic. To represent it in a way that business people can also understand that logic. So that teams can work better together without business people needing to understand code.
So I thought would it be possible to develop a backend that can feed a frontend (be it a Web or mobile app or an integration into something else) without getting into the temptation to develop the frontend with it also.
I wanted a separate frontend (python flask in this case) to demonstrate this. I explicitly used the integrated dashboard in NR to provide an admin tool for something like customer support that have a far less restrictive access to data than any enduser.
My background is a developer that has worked on many Web-based applications where there was always a disconnect between developers and business. So this is an attempt to build a bridge between these two worlds!
I'm afraid that just isn't true at all. Lots of people using NR for all manner of processing. For example, one of our vendors is using Node-RED to help with ETL tasks as part of their comprehensive log analytics.
Well, your concept is certainly sound. But without wishing to blow my own trumpet - I created a front-end for Node-RED years ago that is still in very active development.
uibuilder lets you use standard web development tooling to create your front-end. It also creates a websocket link between Node-RED back-end and the front-end web app. No need for Python/Flask. Node-RED already has ExpressJS built-in and you can either reuse the existing Node-RED user-facing Express instance with uibuilder or you can have it create a new one for you if you need different settings.
The http-in/response nodes exist for this very reason, node-red runs on express, probably the most used library for api's. And with the http-in/response node, you create an API endpoint in literally 10 seconds ? I use this all the time, specifically for this reason. And the front-end you can create with the same nodes.
That NR gets used for many purposes is wonderful to hear but is it used to develop Web apps or do most people just setup a rails/flask stacks on aws and be done with it? At least that's seems the case to me.
And this is not about building a new fronted to NR, the frontend is the product (in this case, an auction platform). It is about putting the logic for an auction platform in NR and building different frontends on top of the logic embedded in NR.
OK, I finally understand what I am trying to say: I'm using NR as logic engine to supply various frontends with data based on the business logic defined in NR.
At the same time, that business logic (because of how NR represents that logic, ie via nodes) business people can understand that logic and potential change that logic without developers having to do any coding.
That's what I found so fascinating about NR: making logic understandable for everyone involved in product development.
Personally, I've never heard of anyone using rails/flask with Node-RED as a back end. Of course, I certainly don't know everything that everyone does with it. But I do know that Dashboard, uibuilder, http-in/-out, etc ARE regularly used to produce working UI's/web-apps.
Of course. Node-RED has always been the back-end to lots of projects with front-ends built using the tools already mentioned, or even Telegram bots and hardware displays. Node-RED provides the ETL and bulk of the logic.
Yes, that is indeed what many of us are doing with Node-RED. Of course, you sometimes also end up with some logic in the front-end too where that makes sense. All perfectly possible with Node-RED and some of its many excellent contributed nodes.
This is exactly why I wrote uibuilder in the first place. Whereas Dashboard provides a really quick-to-market front-end with minimal or zero coding but has eventual limitations because of that. uibuilder started from the opposite viewpoint - provide some "glue" to easily link the power of the Node-RED back-end with the power of web front-ends - both with and without additional support from front-end libraries and frameworks. Over the years, uibuilder has started to add low-/no-code options as well but always keeping to its value proposition of simply making it ever easier to create data-driven web UI's without lock-in to a specific front-end framework.
Yes, absolutely. Though even IT folk are recognising the power of Node-RED to very quickly add logic and ETL processing either stand-alone or combined with other services, systems and applications. Siemens, Hitachi and Atos are 3 big tech companies that I'm aware of using Node-RED commercially for example, merged in as expansions to other products and allowing both technical and less technical users to add logic easily and quickly.
No this is not what I meant, what I meant was that many developers would just use rails/flask and never think of using NR. These frameworks are considered all encompassing including everything you need: models, views, controllers, migrations, templates, specific workflows etc.
What I like about NR is that it allows developers to rethink that paradigm and consider alternative approach to using those frameworks. For me, these frameworks have one common difficulty: the representation of the business logic for product people. Other than combing through the codebase, it is hard for product people to get a feeling for where the logic is and what it's doing.
That's why things like UML exist and many projects to convert UML to code and vice versa. UML has the problem that it becomes "stale" - i.e. the code no longer matches the diagrams and no one maintains diagrams etc. For me then, NR becomes that interactive "UML diagram" (NR is obviously not UML!) that can be modified and immediately the code is changed: diagram and code remain in sync.
I didn't know about uibuilder and yes that is exactly fulfilling the role of my python/flask code. uibuilder is used to build the product just as the python/flask code in my project.
There is one difference though, uibuilder is integrated into NR. My aim was a strict physical separation between "frontend" (product frontend, enduser frontend) and "backend" (business logic, database) codebases. This helped me to better construct a "product API", i.e., an API not for general communication between front and backend, rather a API that represents the product logic that needed to cover all interactions that an enduser has with the product.
This "product API" can then, in turn, be reused for other "frontends" (mobile app for the product for example) as it provides all the necessary functionality.
A contrived example to bring this all together. Taking the product that I built: an auction platform. Lets assume that a business person decides that there should be a minimum bid amount of X dollars. As a developer, I can enforce this business logic in the frontend (i.e. view layer - as an js check) or in the server (i.e. controller level - ruby/python logic) or in the database (i.e. model level as a before insert trigger). Or in multiple places if I like.
Now let's say that the business people decide "Ok, we want to change the minimum bid amount to Y". Then I as a developer (since that limit is in different places in the codebase) need to change every place where X appears and replace it with Y (hopefully I had defined a single global variable!).
What I would prefer (since as a developer, I am lazy) is that the business people can view the business logic, find the spot where X is defined and replace it with Y. And I never have to know that those business people made that change.
And that's what I see NR for and this project was a demonstration of trying to do that. Additionally, as the logic is in NR, those changes are also reflected in the "product API" (as it comes from NR) so that all clients are immediately changed.
Of course this is a contrived example and after the third time readjusting the minimum amount, there would be some interface for the business people to do that on their own. But one can assume that business will continually be adjusting something and the more they can adjust on their own, the more I can enjoy my laziness (as a developer)!
Understood. That is why sensible organisations have standards They should guide developers on their software stack and workflow. In this case, the standards should indicate when and where it is appropriate to use NR in the stack.
And this represents a training need possibly for the developers.
As long as you recognise that the same disciplines need to be applied as for any other development. That documentation is critical - while some flows are fairly obvious, many are not and still need some level of documentation - though of course, much/most of that can be done in the flow itself.
As for traditional development, any decent code workflow should have both top-down and bottom-up documentation. Top-down showing the architectural overview - main components, interfaces, etc. Bottom-up often being auto-generated as part of CI based on code comments which should also be subject to standards so that things are properly documented. As you say, something like UML is very hard to keep up-to-date since it is somewhere in the middle, fairly detailed but hard to keep current. More important to keep updated are things like interface standards and data schema's and the schema's at least could possibly be auto-generated.
It is a good point. In fact though, it IS possible to split the front- and back-end when using uibuilder. It means that you may have to use the uibuilder.start(....) function and include the additional options that tell the front-end client where the node-red server is and defines the name of the private websocket comms channel. Normally, the client works this out for itself but it can't always do that if you are serving the code from a different web server. So it is absolutely possible to use the uibuilder client library as simply an add-on library in your front-end code.
Whilst working that way reduces the overall benefits uibuilder gives you - in that you might not be using the library manager for example. It still supports all of the other features: The websocket comms channel (with fallback to other methods where websockets fails to connect and with auto-reconnect), The ease of sending messages from and to Node-RED, automatic control messages that informs your Node-RED flows about clients connecting (so you can send cached data for example and make security decisions if needed), disconnecting, pages becoming visible/hidden, etc.
Perfectly reasonable. Though I might ad a reminder that uibuilder itself is an API.
Personally, the idea of allowing the business potentially unrestricted access to easily change the logic of my auction site would be a recipe for disaster! As an IT Enterprise Architect, I would have some harsh words for anyone who came to me with a service idea that allowed that.
But that is a whole other conversation, lets not go down that rabbit hole here.
I think this partly why we are having this discussion, because of our different backgrounds. I come from the startup - everything needs to be done yesterday, no documentation, no standards, release early, release broken, fix later - world.
There as a developer you are pleased when business people can fix their own issues without coming to you and asking why you haven't made that change that they requested five minutes ago! My experience from that world is that clarity/knowledge is super important but no one has time to invest into it. (How many attempts have I seen of people setting up a wiki or google docs or ... all of which simply decayed overtime because no one used/maintain/found the documentation.)
In that world, having a tool whereby you can code, show business people their logic and have up-to-date documentation, all in one, is a godsend.
Following our conversation, I made an update to the project readme to clarify exactly what we having been discussing:
I tried to create a project to demonstrate this usage of Node-Red as a "UML-like business logic engine". That Node-Red was not invented to scratch that itch is clear, however it can be misused to scratch that itch!
I am misusing NR and I know that
If someone else also took this approach to use NR as a UML-like-design-tool, then their architecture (e.g. to use uibuilder or external own codebase) would be most likely different but that's the fun part, discussing the alternative approaches to inventing the wheel!
And of course I don't have to use NR in this way, in fact no one has to but - and here NR is at fault - you can use NR in that way, so why not try!
It's like when fire was invented: no one imagined it would be used to send rockets into space, those who invented fire, just wanted to keep warm!
Nor perhaps did they think - for about the first hour I imagine - about using it to burn down their neighbours house! But that is what some startups are creating without thought of the consequences.
But hey, I certainly get your point and don't disagree that NR can be used in many different ways including the way you describe. And it isn't a terrible approach all things considered. Until your business user completely borks the system one day that is - some automatic backup and easy recovery is probably in order.
But the main point I wanted to get across was that there are already options that are widely used that are suited to many use-cases. That point was made so people in the future can look at this thread and not be misled. I wish you all the best in your endeavours.
Enjoyed reading the exchange above @gregorius your enthusiasm about node-red making the business logic easier to follow for non-techie folks echos how I felt when I played with node-red for back-end json apis a few years back.
My question: I didn't wind up pursuing node-red for my ("enterprise") use case at that time largely cause I didn't get the impression it was designed with the kind of multiple-server-cluster-deployed-to-aws-with-flows-reliably-stored-in-a-database-etc. type of support needed by a big company worried about security and scalability and 24x7 guaranteed uptime.
I'm wondering specifically for your @gregorius back-end (or @TotallyInformation do you know if other people in general):
are serving node-red http apis on a cluster?
are doing so on aws or some other cloud provider (maybe with docker?)
have their flows securely stored ideally in a database or maybe using the github "projects" feature? (as opposed to that one flows.json flat file if I recall?)
and are they able to deploy flow changes to each server instance as nicely as with a single node-red instance? without a traditonal "build and deploy the files" kind of step?
The github repository is managed throughout the "project" feature of NodeRed. Then, when changes are made, a new docker image is made with the updated set of flows.
So, upgrading the flows is made by an upgrade of the docker container version.
The parameters (env variables) used within the various nodes are initialised using docker compose.
All this is currently deployed on an on-premise openstack cluster. Using ansible you can easily update all the docker-compose and then, done.
I've been thinking about scalability but haven't gone much further than theoretical thought games. At the moment I host everything on Heroku with two apps - one node-red and the other web-frontend. Deployment is git push for the web-frontend, for Node-Red is replacing the entire flow and using the deploy button - not optimal but within the current pain limits.
Web-frontend is stateless frontend that uses a stateless API (supplied by node-red) so scaling that isn't particular problematic. (More precisely: state of the web-frontend is stored on the python server side, python communicates with the frontend and the frontend communicates directly with Node-Red through the web sockets. Python instances share state via Redis.)
Within my Node-Red setup, there are three components:
stateless but authenticated API which powers the frontend
web socket endpoint for the frontend
admin tools for the back office parts (customer service etc)
Theoretically each component could be a separate cluster. However the "link in/out" nodes would have to become http-request nodes. Socket notifications are partly triggered by my-custom-API flows, these triggers are link-in/out since it's all within one server, one flow. However if the socket part is a separate service/cluster, then these triggers would need to become http-in/out nodes (or better said: additional triggers would be needed).
Which leaves the stateful postgres database upon which node-red is built. All three components communicate with the database. Having a read-only-follower of the postgres database would allow the web-socket component to access the read-only follower - sockets make no updates to the database. But once again, it comes down to the database and how robust it is. I could then begin with sharding and other DB optimisations but far too early for that.
Overall though, I still have the feeling that in the comparison between the question of Node-Red scalability versus the advantage of business-flow clarity, the clarity is more important than scalability. Most companies go on about communication within the company and not about the scalability of their software - there is a reason for that!
Question: Is there a way of linking several instances of Node-Red together (i.e. a shared MQTT instance or something)? I was just thinking about redesigning all the link-in/out nodes in my flow and began to have a headache - hence it would be a great help if link-in/out nodes where "addressable", i.e. work within a cluster of node-red instances, with each node-red instance having it's own responsibilities.
link-in/-out are internal only, and I think could only ever be that given the design.
But Node-RED excels at communications so you could use almost anything to link between Node-RED instances. Typically MQTT would be used because it is robust and simple. But raw TCP or even UDP could also be used if needed.
The only thing to watch out for is serialisation. To get a JavaScript object to output to MQTT (or pretty much anything else), you have to serialise the data and there are some objects that cannot be serialised. Normally won't impact you but when it does it is mighty confusing at first. The main example are the msg.req and msg.res objects that ExpressJS can add - for example from an http-in or an http-request node. These would have to be removed - I think that the mqtt-out node does this automatically.
Node-RED as a back-end... Node-RED IS a back-end as it is a web server for the dashboard. i.e. the Front-End would be the dashboard. You can use multiple instances (installs) to do tasks that are unrelated, and perhaps on different servers, then bring the data to a main instance. We have instances which are running on Windows Server that handle data from other "nodes" and TCP devices and MQTT, of course. The MQTT broker is on a separate Windows Server with SQL Server to store data from all systems.
Link nodes cannot be addressable between instances. You can do this with MQTT and you still need to serialize or name the connections. You can also do this with TCP nodes. That has much more overhead as you need to maintain connections.
What your looking to do is reinvent the wheel with the same wheel. What I would suggest is working with Node-RED as it is (Which is pretty dang great), and get familiar with what it is and does, then you can do amazing things.
What I would suggest is working with Node-RED as it is (Which is pretty dang great), and get familiar with what it is and does, then you can do amazing things.
Right and what if I don't have a Windows server and IoT devices? Am I then not allowed to play around with Node-RED? Should I find something else to play with?
You know what I using Node-RED now for? As a mind map tool! Yes, mind maps - incredible. Each node in my mind map is executable so that I can generate a markdown document of various nodes and connections of my mind map. And you know that documentation tab on each node - I am using that not for documentation but for texts explaining my mind map nodes. Yes: I am misusing Node-RED - again.
I think a lot of people here think Node-RED is a hammer with which you can only hammer in nails. Ironically a hammer can also be used to hit thumbs and break windows, may be that's what I'm happily doing without being told that I can't do that. Or does it not count for "amazing things" when I use it for something other then pretty dashboards and turning lights on and off?
As I said before: it's all Node-REDs fault for being so flexible, although upon reflection ... I am also partly to blame for being imaginative enough to use that flexibility.
I guess what I should be saying is that I'm using Node-RED as "backend modelling tool with builtin execution of the designed backend" - but that just doesn't sound catchy.
Btw why isn't there a Node-RED icon in the icon collection? I had to use the registered trademark icon ... does that make Node-RED non-turing complete
Hmm, well just because you have a nice hammer, it doesn't mean that you should use it for everything.
I might be able to hammer in a screw but it will probably make a mess.
Of course, you are perfectly at liberty to use Node-RED however you like. But it has a set of primary purposes that are being widely used and also has plenty of outstanding changes on the backlog that I suspect most people would consider as higher priority than enhancing its drawing capabilities - not that we wouldn't also like some of those as well.
Also of course, if the core devs were to agree, you could consider some code submissions because resources for change are always at a premium.
A reasonable point. Of course, you can use your own images as well. I recently updated the uibuilder nodes to use custom icons using SVG images.
I spent many years building and talking about backends and many developers seem to enjoy the tapping of keys rather than the clicking of mice!
