PYNODE RED async message responses

General
1

hello
I am using successfully pynodered (Python node red) to do a lot of processing in the background for all my node-red needs.
Now I am trying to add "async" asynchronous functionality to my backend system via pynodered.

in general, node.send via NODERED works for asynchronous message responses.
But i dont think pynodered supports this.

has anyone tried sneding asynchronous message responses back to node-red from Python?
any examples / code references will be appreciated.

thanks

2

Looks to me as though this creates a simple TCP server (or maybe UDB or UNIX Pipes, I've not read the code). Your Python function is exported to update some JavaScript files that provide an automatic link. That's why you have to restart both your Python code and Node-RED if you make changes.

Fun but a little over-complex perhaps since you could actually use a single Python server with a single connection to a single node and get much of the same effect. But of course, that would require more programming knowledge from the users. Personally, I think I would have created a second channel in the Python server that output a list of available functions and then the node in Node-RED could request that list when you open the configuration UI in the Editor - that would save Node-RED from having to be reloaded each time.

But I digress from your question.

I see that Flask is being used for the Python server which is good since that is non-blocking and async I believe (my Python is a bit rusty).

However, I cannot see any mechanism on the Python side that lets the server await a response from Node-RED. You would need this for what you've asked for.

Fundamentally, Python and Node-RED (being a node.js application) are very different in handling async processing. Node.js is very much async driven and Python is not (without 3rd-party libraries).

So unless you can program some way, on the Python side, to wait for a response (along with all the checks you need to prevent your server from being blocked), I think you are rather stuck.

Hmm, one rather annoying thing about the pynodered code is that they don't allow issues to be raised in GitHub so it is rather difficult to feed back any suggestions for improvements.

If your coding skills are up to it, I would create my own Python server that acts as an API server. You can then interact with it as a REST API using the core request node in Node-RED. That will let you GET data from your Python server and POST data to it for example.

1 Like
3

thank you @TotallyInformation - appreciate the detailed response.
I think I can brush up my coding skills.
So PynodeRED does not do what I need.
NODE RED itself allows sending asynchronous messages via node.send followed by node.done
This is what I wanted to acheive.

Regarding your comment:

Fun but a little over-complex perhaps since you could actually use a single Python server with a single connection to a single node and get much of the same effect.

Can you elaborate how would I go about implementing the python modules, should I choose to accept this mission? :slight_smile:
in my python code, all I want is an ability talk to node-red server running on my localhost 1880.

Currently via PynodeRED I can grab the entire msg object, and node handle. (which I incorrectly thought was going to help me do node.send and node.done) but this so called "node" object from pynodered is not the one I can call these underlying core functions.

TIA
shan

4

I meant to ask for what will the top level flow in my python look like? not actual code. :slight_smile:
Also, can Python module (which I have written for normal socket programming) communicate to node-red server on local host :1880
If so, where is the messaging protocol defined for NodeRED?
thanks

5

Would it not be smart to use MQTT for the Python Server <---> NR communication.

Fast, reliable, easy on both ends. NR has already built in support for MQTT. Adding MQTT to any Python code is dead simple too

6

thanks. I have added MQTT /implemented RabbitMQ completely for the project.
I can reuse that for Node RED too.

My question is:
-from my Python code, if I send MQTT commands/packets, will Node RED accept it? What format do I need to send them?
Where is the detailed documentation for MQTT support of node red?

appreciate the discussion.

7

Two parts.

  1. Python server

    Use Flask or something similar to create a Python-based web server. That server will serve as the conduit between Node-RED and Python. Start with a simple REST server that allows data to be fetched from Python (probably via simple GET commands from the NR request node). Then you can elaborate that with the ability to push from NR (probably via POST commands with a JSON payload).

    In Flask, you will create some breakouts that respond to specific parameters that you will use in NR. for example. If your Python server is running on port 1889 on the same device as NR, a request node issuing GET http://localhost:1889/?fn=lowercase&val=SomeMixedUpText. You use the fn parameter to direct Python to the appropriate function and pass any other parameters according to need.

    Using a POST lets you get more complex.

    Always return the data as JSON and you can easily then convert it to a payload on the NR side.

    A simple but effective REST server that can be expanded however you like with Python functions, classes and modules. Link to NumPy, etc to get structured data analysis for example. Just watch out for how much data you pass back and forth, don't let it get too big.

    To go further and expand performance, consider implementing a websocket connection. This is easy on the NR side and you'll find a library on the Python side too.

    With all of this, you still have the option of expanding your setup so that the Python server is on a different device to the NR server should you need to.

    If you need to exchange really big data, you will need to go further and look at streaming or chunking but this should be enough for many uses.

    You can, of course, implement something similar in reverse. In fact this is even easier since Node-RED already sets up all the servers you need and makes them accessible. So you can call your Node-RED REST server from your Python code. Again, using a suitable request library for simple REST API's or extending to websockets if you want something a bit more dynamic.

  2. Node-RED Node

    For node-red, you really only need the request node to make the calls to your python server.

    If you wanted to get clever and productionise this for other users though, you could create another REST API in Python that you configure to return a list of the various functions users can call, perhaps also returning info about the parameters they need.

    Then you could write a node that, when you are configuring it in the NR Editor, before it shows the UI, it does a REST query and gets back the list which the user could then chose from in a drop-down. The API could even return some hints on how the node could validate the inputs.

    Several existing nodes do something like this. The serial nodes and uibuilder for example.

I should add that I had been considering doing something like this myself as I needed to do some data analysis on some largish CSV files recently and this is an area where Python really excels. I ran out of time and had to use a nasty combination of Excel and bits of stuff written in a Jupyter notebook as a one-off instead of something repeatable.

1 Like
8

The top level would be a simple Python web server. Probably with a big switch statement that called out to other modules as you need to.

Well, if you already have something with either websockets or Unix pipes, then you are already mostly there. NR excels at websockets. Use the websocket-in/-out nodes.

You are over-thinking it. Just exchange serialised JSON and you are good to go with pretty much any data you want (well, anything that can be serialised JSON anyway!).

Yes, you can use that instead. But a direct connection is likely more efficient in general. Just depends what you want to do and how fast you need it to happen. Also depends on what networking might impose on you. MQTT is useful for an extended network but for a simple local network, websockets will be more efficient.

A REST server is not as dynamic but translates well and is easily secured no matter what network you may have, including the Internet.

Again, no need to over-think things. Node-RED will handle anything you want to exchange. From simple text/numeric/boolean values to serialised JSON. Or anything else you like really as long as you can encode it on one side and decode on the other.

In the documentation for the mqtt-in/-out nodes. Mostly, it "just works".

If you do use MQTT, the principles are the same as I've already described. Set up a Python server that lets you publish and subscribe from your own code modules/classes and keeps running in the background. That is if you need 2-way comms. e.g. you want to drive from Node-RED.

I don't know enough about MQTT libraries for Python to be able to say whether that will be especially effective. You will really still want to be able to implement an async service.

1 Like
9

Flask is very, very sync. I love it, but it’s sync and blocking. If I need async I use Quart, a framework inspired by Flask and API compatible as much as possible, but async.

2 Likes
10

Ah, thanks Lena, I didn't know that. Long time since I did much serious Python.

1 Like
11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.