Hi,
I am doing stream analytics using node-red with MQTT and data is coming form OPC.
On flow start: Using inject function will minimum time I will initialize common functions, objects, variables, configurations needed for processing the data coming form OPC through MQTT node.
First layer: I have couple of flows with 400-500 MQTT nodes.
Second layer: Each MQTT output goes to one function node.(Having one for each to add meta data to the message like type of message and index if I am using array to store them etc).
Third: Will connect to couple of other common functions to update objects depending on the grouping of nodes using objects, arrays and variables and conversion data types or at times ignoring few values.
Fourth: Will connect to couple of other common functions to process those objects and determine the state of the system.
Fifth: Will go to one single function node and form there to API, logs, debug node etc.
This is the basic strait forward architecture I am using.
NOTE: Looking for all kind of inputs on this especially expecting on performance, memory, speed maintenance etc.
Q1: Does the above architecture makes any sense or Do any one feel few changes need to be made?
I have been looking into this for couple of weeks now how many other systems do this.
Saw that common big data application like Spark stream and others do the state capturing using queries and few uses databases and in memory(in spark) for storing data like context in node-red.
Now not sure can they do the ignoring few values, conversions, to object grouping(sure they are not doing this because they use queries).
Q2. Does they do this or its unique to node-red?
More over I need to save current and also previous values too so that I can determine the change in value or it is increasing or not etc and have very custom logic for processing data like keeping previous captured states, value averages etc.
Q3. Does they do this or its unique to node-red?
I think this is where for me node-red is on greater hand apart form having beautiful way of implementing complex stuff using UI and zero learning curve because I am familiar with node-js and java-script.
Q4. Are my assumptions sounds good or dumb?
Any inputs form this community will be a great help for me to do stuff more efficiently and effectively with clear mind with no confusion and doubts on what I am doing and implementing.
Just for information my current node-red stats:
{
"TOTAL":2358,
"CassandraDatabase":1,
"cassandra":1,
"catch":12,
"change":1,
"comment":300,
"debug":102,
"delay":19,
"file":37,
"file in":1,
"function":1025,
"http request":44,
"inject":72,
"json":1,
"mqtt in":712,
"mqtt-broker":12,
"switch":2,
"tab":15,
"ui_base":1
}
With regards,
Nithin B.
