I’m building a project called Anvoie, where AI agents (what I call digital envoys) screen potential social or dating candidates on behalf of the user. Instead of endlessly swiping or scrolling, the user delegates the initial screening to an AI that learns their preferences and filters profiles before they ever see them.
Because of that architecture, the system relies heavily on asynchronous jobs. Currently I manage these using BullMQ with a Node.js worker. While this works well from a technical standpoint, it’s starting to become difficult to reason about all the queues, workers, and job flows as the system grows.
I’m considering moving part of this orchestration to Node-RED because the visual representation of flows could make it much easier to understand and manage the asynchronous pipelines.
Has anyone here used Node-RED as a visual orchestration layer for queue-based architectures like this?
I’m curious about:
Whether Node-RED works well alongside existing queue systems like BullMQ
How people structure complex asynchronous workflows visually
Any limitations I should be aware of before attempting this approach
Do they date themselves? I mean why even have humans in the loop. I love the idea of a dating platform where AI agents date each other. Humans can then become digital voyeurs.
No, they don't date themselves. Envoys do the initial part of the Funnel so that their Users don't need to waste time on their phones. Once an Envoy finds possible matches based on initial hard filters, it starts conversations with envoys that belong to users that passed the initial filter. Sometimes during conversations, an Envoy asks for things that it may not have in its context about its User, so it contacts its user and wait for its answer. User can answer envoy requests using the platform, or via Instagram. When envoys finish talking, the conversation is evauated based on some criteria configured by the User. If the score is bigger than what was configured in filters, a Match is created. Both users must accept a Match to let Anvoie connect them. When they match, an e-mail or telegram message is sent out with their e-mail or phone number.
Sorry I will also provide my "technical" insights:
I think Node-RED is perfect tool for organising queues and handling messages. Visually that's what your looking: data flows through computation which is basically a message bus or queue. RabbitMQ is well integrated in Node-RED, so it all shouldn't be much of a problem.
I do this per-flow-tab and store each flow tab as a separate file, combining flow tabs for a single system. I.e. that's my flowhub.org works. I rarely use sub-flows because they are harder to extract.
What that looks like can be seen over at Browser-RED:
That is my library of flow tabs each categories and can be included by double clicking on them. These are all hosted over at GitHub, so that I can change them inside Node-RED and then push them back to GitHub, without leaving NR.
For each project, I have a separate collection of flows and each collection is stored to a separate public or private repo at GitHub. And each collection is linked to a api key so that I can seamlessly switch between collections.
The management of collections and api keys is done under the push tab:
Those are the projects I work on and each has it's own repo with flows.
I don't store the entire flow to flowhub, instead I have a github backup flow that takes the entire flow and stores it to a github repo. And only updates get pushed, i.e. the backup flow runs every 15 mins and if nothing is changed, no update is pushed to github.
Yes, it does get complicated. My approach to structuring flows is that flows ...
... have clear entry points (be it a MQTT in or HTTP in node for example). adding inject nodes to demonstrate flows is also good. Exit points aren't that important because there can be many but you can see them.
... have good documentation include of any ENV variables required to run them.
There is a clear contract between flows, i.e., "this flow handles kafka messages of this type ...". By "contract" I mean boundaries along message bus (for example) boundaries.
I avoid having too many dependencies between flows, I prefer to copy & pasted functionality than having a single common flow to provide functionality - but that's a matter of taste.
Node-RED is great in linking flows together using link-nodes, i.e. if I have a link-out node that references a link-in node on another flow, then those nodes will be linked as soon as I've imported both flows. The linkage will be stored in both flows and will "automagically" appear if both flows are available in the same NR instance.
I put as much functionality into a flow as I can. Probably the best why to explain this, each flow is a class not a method/function - i.e. if I had a class and a bunch of methods, then a flow won't be a method or function, but a class. On the other hand, a flow won't be a complete system (unless it's a small system ;)). I.e. I have a granularity which sometimes gets out of hand.
Having said all that and currently working with a project with 2.5 million lines of textual code, I would not want to visual those 2.5 million lines of code. Which is a pity because the project can definitely use a breath of fresh air but the complexity and interdependency of that code is left best as is!
Just watch out for those pesky escapes that AI Agents are already becoming known for! I've seen several notable examples recently: Deleting drives, empyting a mailbox and just today, one decided to start a sideline by crypto-mining!
I foresee some "interesting" dates being created! Should be fun.
Agree with these. Really, in a msg q based system, I think that each q'd message needs to be a stand-alone process. Anything else becomes quickly unmanageable. Worse, in a Message bus arrangement, you also have to start managing replies and rejections and will need to create a dead message q as well. (For context, I did some work many years ago on the NHS electronic prescriptions service - they were loosing 1% of prescription messages and they thought it was a "good" "low" number. It wasn't! Given that it was already handling over a million prescriptions per day - that was a lot of people not getting prescriptions!).
