Cant get array buffer refrence to work... Help please?

[Dyslexicbloke]

Hi folks,
very new to this, Node-Red and JavaScript, but I have been programming, one way or another, for years.

I am trying to build a function node to process Modbus input to get, optionally, Uint or Float/Real from the 16 Bit word array, or 8 Bit raw buffer, that the Modbus getter returns.

I did a little digging and found that the most efficient method was to use Typed Arrays and change the view of the underlying buffer to get an appropriate value.

Not something I have come across before but simple enough conceptually, letting the API/script do the heavy lifting.

There are several methods/approaches but in essence I need access to a buffer, to copy part of it to a new typed array.

This dosnt work...

var ArrayFomMsg = [65535,200,300,400];       // This could be very long
var strStat = "";

var ComBuf = new ArrayBuffer(4);             // Define a common buffer, big enough for a 32 bit number

var InArray = new Uint16Array(ComBuf);       // Define input array with the new buffer
InArray = ArrayFomMsg.slice(0,1);            // Get the first value (two Bytes)

var OutArray = new Int16Array(ComBuf)        // Define second array with the same underlying buffer

strStat = InArray[0].toString();
strStat = strStat + " > "
strStat = strStat + OutArray[0].toString();  // View that as an Int

node.status({text:strStat});

This does..

var ArrayFomMsg = [65535,200,300,400];           // This could be very long
var strStat = "";

//var ComBuf = new ArrayBuffer(4);               // Define a common buffer, big enough for a 32 bit number

var InArray = new Uint16Array(1);                // Define input array with the new buffer
// var InArray = new Uint16Array(ComBuf);        // This also wofrks and ComBuf dosnt need to be dimensioned!
InArray = ArrayFomMsg.slice(0,1);                // Get the first value (two Bytes)

var OutArray = new Int16Array(InArray)           // define second array with the same underlying buffer ?
//var OutArray = new Int16Array(InArray.buffer)  // Should work but doesnt - errors only if array is accessed 

strStat = InArray[0].toString();
strStat = strStat + " > "
strStat = strStat + OutArray[0].toString();          // View that as an Int

node.status({text:strStat});

If I define ComBuf too small, or even not at all, it still works.
I realize I could just define InArray(1) but that doesn't help, although it to still works
The slice is working because trying to access OutArray[1] fails.

So..
If all I want to do is look at a 16 Bit input as a 16 Bit view all is well.
However I want to be able to take 2 words of a 16 Bit array or 4 Bytes of a raw buffer and get either a 32 bit view, Int, Uint or float, whatever.

Many hours of fiddling and bizarrely this works...

var ArrayFomMsg = [255,255,300,400];           // This could be very long
var strStat = "";

//var ComBuf = new ArrayBuffer(4);             // Define a common buffer, big enough for a 32 bit number

var InArray = new Uint8Array(ArrayFomMsg,0,2); // Define input array from first two values (two Bytes)

var OutArray = new Int16Array(InArray.buffer)  // Define second array with the same underlying buffer

strStat = InArray[0].toString() + ", " + InArray[1].toString();
strStat = strStat + " > "
strStat = strStat + OutArray[0].toString();    // View that as an Int

node.status({text:strStat});

The issue is that I don't understand why.
The def for InArray should be based on The buffer of ArrayFomMsg
new Uint8Array(ArrayFomMsg.buffer,0,2) fails
However the reference provisioned by defining OutArray on the buffer of InArray is fine and works.

I am utterly confused,
Can anyone explain this and tell me what I am missing.

The array object in the/a message works OK with x.split method or the syntax that extracts a new array form an existing one when defining it.
(Sorry I don't know how that structure is referred to.)

BTW
The object in the message from the FlexGetter called 'buffer' is actually an array and its endness, is that even a word, is reversed with respect to the endness of the words relative to a 32Bit, I think.
My head is mashed right now!
Not that it matters, swapping bytes/words about before looking at the new view isn't a big deal I guess.

Is there a JS method to exchange array elements that avoids manually doing two copies?

I really want to understand this .... It seems fundamental and I hate not having a handle on it, especially when most of the, 'standard' examples don't work here.

Long firs post, sorry

Looking forward to being corrected,
Cheers,
Al

[bakman2]

Welcome. I am not familiar with the data that comes from modbus, but there are modbus nodes available that can handle the problem you are trying to solve.

To install, right top hamburger menu > manage palette > install tab > search modbus

node-red-contrib-modbus

[Dyslexicbloke]

I am using that lib...
As far as I am aware, and to be fair I am not aware of much the filter modbus response would do that.

Unfortunately it wants a file, iofile, which it seems happy to make but I cant find where it puts it or what should be in it.
I did tell it to create one, an iofile, and it prevented Node-Red processing, coming up with an exception, something about not finding an extension I think.
It took me hours to figure out what I had done
Removing the file, which required a node that was using it to be on a flow, more hours, fixed the crashing.

I am not suggesting it doesnt work just that I don't know how to configure it.

On top of that it look as though each filter is looking at a specific offset in the data and as they only have a single output I would need split nodes for each one...
Of course I could be, in fact probably an, entirely wrong...

This is what I am doing right now, well trying to do.
right now the conversion seems to work but the offset inst extracting from the correct place
just debugging that now.
That is probably a bit grand, fiddling with it would be a better explanation
Its light in the UK now and its Sunday, I probably need more beer, I am sure that will help!

In other news... I will probably want to do the same with other data, MQTT and the like, so a consistent approach seems attractive.
It is also a good learning opportunity.

Al

[bakman2]

Ah ok (as I said I am not familiar with it) I thought those node would handle the output.

Did you search the forum, because you can't be the only one, then again that is also an assumption

[Dyslexicbloke]

I did...
I was looking specifically for the iofile thingy as I got everything else working quite quickly.

I think endness is in the getter/putter so the 16bit Ints are good but that doesn't help with Uints or Floats

processing the values inst hard, even longhand, although floats are a bit of a phaff, it is the indexing/splitting of the array, well the array buffer, I cant get right.

If I stick to reading ints and use the split method on the message array It will work but that is only half an answer and thus somewhat unsatisfying...

Buffer manipulation should work, one way or another, or at least if it doesn't there must be a good reason that is going to bite me again if I don't know about it.

That said I will take what I can get help wise and be glad of it.

Cheers
Al

[Dyslexicbloke]

OK, so this is the plan...
I want to read a block of memory, that may not necessarily contain variable of the same type in a block.
This is because, well stuff just doesn't, floats are fairly rare on modulus kit but but Ints, as oppose to the native Uint 16 Bit word are and doubles, signed and unsigned crop up fairly regularly too.

On my own PLC/s I can easily organize things in sensible blocks and avoid floats by scaling 16, or even 32, bit Ints o Uints.

But as I said, conversion isn't the problem, its the array handling that will get and set stuff.

The rational is to build a flow like this...

msg in containing an array, potentially a big one.
Each function not references that array, not copies it, and pulls out a child array.
Once the node has a small array javaScript provides, or at least should provide, a simple way to view the data in it.

TypedArrays seem to work like arrays with implicit methods, returning formatted values from base data

8 Bit 255,255 is just that, 255 twice
If you stuff into a typed array 8 Bit as two you will get two elements in the buffer, which is the memory underlying array I think.
Array8[0] will be 255 as will Array8[1]
However if you then look at it with a second typed array, say 16 Bit Int, defined with the same buffer
the same two bytes will represent -1 in Array16[0]

It gets a bit fiddly deciding what to put where and how, making it al 8 Bit Uint first would be safest, but it works, at least my limited testing suggests it works in Node Red as I have implemented it.
I may have interpreted this all wrong but the implementation works however badly I understand it.

My problem is the split() because it will not get me to the bytes, at least not for everything.
For the Modbus node I have the bytes as an array, all be it called and labelled buffer.
(I am unclear about the distinction at this point but they support very different methods)

It isnt just about getting or not getting date, it is as much about getting my head in the game and doing that well seldom involves the easy option.

Now if I missed something fundamental bout Node-Red and shouldn't be doing this at all for this task I will happily learn but it isnt going to stop me wanting to understand the array issue, irrispective of if the issue is mine or something different about JS in Node-Red

I hope that explains my stance... But willing to learn the correct /preferred way whilst still learning he detail probably sums it up.

Cheers
Al

[Steve-Mcl]

Hi @Dyslexicbloke, I haven't read all your extensive and detailed posts (so I apologise up front) but based on this part...

@Dyslexicbloke

I am trying to build a function node to process Modbus input to get, optionally, Uint or Float/Real from the 16 Bit word array, or 8 Bit raw buffer, that the Modbus getter returns

Have you considered just using the handy helper functions the buffer object provides e.g....

let value1 = msg.payload.readFloatBE(0);
let value2 = msg.payload.readInt16BE(2)

If I'm not mistaken, the modbus nodes return PLCs values in a buffer?

[Dyslexicbloke]

Nope... But that would be ignorance not arrogance.
Would ye happen to know where the documentation for that would be, I didn't come across it and it wasn't for the want of trying.

let value1 = msg.payload.readFloatBE(0);
let value2 = msg.payload.readInt16BE(2)

I assume that there is an LE version (Little Endian)?
is (0) the index of the source array/buffer?
will it work with an array of Bytes or just a buffer?

Are there reciprocal functions to push bytes, or append bytes, to a buffer/array when given a number?

If the docs tell me all this don't bother answering, I am happy to do my own legwork, I just dont much like walking in circles

'O' and what with the Let... I haven't seen that in use since my BASIC days, I know its optional in many languages but hardly used. Is there some significance here as I know it isnt required for a general assignment.

Thanks
Al

[Steve-Mcl]

@Dyslexicbloke

Would ye happen to know where the documentation for that would be

In the link i created...

@Steve-Mcl

helper functions the buffer object provides e.g....

@Dyslexicbloke

I assume that there is an LE version (Little Endian)?

Absolutely

@Dyslexicbloke

will it work with an array of Bytes or just a buffer?

Just a buffer however, if the array is all bytes, conversion is simple

let buffer = Buffer.from(arrayOfBytes);

@Dyslexicbloke

'O' and what with the Let... I haven't seen that in use since my BASIC

Let is similar to basic but is block scoped.
E.g.

let X=1
{
  let X = 2 //different variable
}
console.log(X}
// 1 

To put it simply, let == "less bugs & better programming"

[Dyslexicbloke]

Cant believe I missed the link, wood for trees and all that.
Buffer,from(Array) Didn't find that either, probably looking and wrong JS version or something, I guess I will learn about that as I go.

I was using Array.buffer in an attempt to get a ref to it., failed...
and split() was kicking my behind because I couldn't get it to work when trying to get a float, odly it worked OK with 16 Bit Uint, not sure why it wouldn't handle the 4 bytes but it wasn't a good solution anyway.

Rebuilding function now, I may even red the documentation if I cant get it to work with your great pointers.

Back in 10 with a bit of luck.

Thanks this looks like my solution.
Best regards,
Al

[Dyslexicbloke]

Steve...
Thanks a bunch, that is way simpler than my previous plan, although it will needs tweaks.

I haven't tested Int32's but they will not work, yet, from this device at least (Schneider M221 PLC)
16 but values are fine using BE.

The float inst working but I swapped the word order in the PLC, just by altering words defined in the same memory space as the float, and that did work in BE

My test flow is below, all the values are correct, notwithstanding float silliness.
NB, this is still pointing at a word values from a float but in the wrong order in the PLC

The trigger sends an empty string for now, the first function sends initial config for the get, which the getter uses.
Output is arrays and buffers, multiple views of them.
My function node is looking at payload.buffer and pulling values from that.
all the nodes are identical but add/modify a byte offset value in the msg to allow the following node to continue reading the block of memory, or not whatever.

It seems to work well... functionally
I am planning to do a similar thing to get values and send them as a block.
I generally don't bother with coils, I map those in the PLC to bits in an int. This approach gives flexibility when it comes to handling errors and offline, com fail, states.

I would very much appreciate comments positive or otherwise, I am new to flow programming, as you can probably tell.

The code in the function is below the screenshot, below that I am asking for advice RE arrays and buffers again... Dam floats.

* 
Takes an buffer and extracts 16 or 32 bit values.
Processing starts at 'Offset' in buffer'
The output is a single value in 'payload.value' on output 1
msg is passed to the remaining flow, output 2, with Offset set for the next value
*/

                                             
var OutType = 1;                                              // 1-INT16 2-Int32 3-Float32
/*************************************************************//***************************************************/

var SrcOffset = msg.offset || 0;                              // Get the current offset = assign 0 if undefined
var ConvRslt = 0;                                             // Init Conversion Result
var NxtOffset = 0;                                            // Init Next Offset
var strStat = " @ " + SrcOffset.toString();                   // Init status string
                                                              
var msgResult = RED.util.cloneMessage(msg);                   // Clone msg Object, for the result on output 1

msgResult.payload = { value:null }                            // Clear the origional payload leaving the rest intact
                                                              // Init with just the one item!

switch(OutType)
{
  case 1:                                                                      
    ConvRslt = msg.payload.buffer.readInt16BE(SrcOffset);                      // Convert the buffeOffset, to Int16 Big Endian
    strStat = "Bytes to Int16BE " + strStat + " > " + ConvRslt.toString();     // Set lengt and status text asociated with output
    NxtOffset = SrcOffset + 2;                                                 // set up for the next node
  break;
  case 2:
    ConvRslt = msg.payload.buffer.readInt32BE(SrcOffset);                      // Convert the buffeOffset, to Int16 Big Endian
    strStat = "Bytes to Int32BE " + strStat + " > " + ConvRslt.toString();
        NxtOffset = SrcOffset + 4;
  break;
    case 3:      
    ConvRslt = msg.payload.buffer.readFloatBE(SrcOffset);                      // Convert the buffeOffset, to Float Big Endian               
    strStat = "Bytes to Float32BE " + strStat + " > " + ConvRslt.toString();   // Not working, byte order OK- Word order reversed
    NxtOffset = SrcOffset + 4;
  break;
}

node.status({text:strStat});                                 // Output the status text Config and value

msg.offset = NxtOffset;                                      // Update the offset for the next node...
                                                             // This will add the property if it dosnt exist
                                                             
msgResult.payload.value = ConvRslt;                          // Update the payload for output 1 with the result                                                        
return [msgResult, msg]                                      // Two messages, 2 has an updated offset value, 1 is the result.

let was giving me errors because it didn't like defining things more than once.
It didn't seen to treat the switch blocks as separate scope's

As I said earlier, the Float conversion needs the word order altered to work.

I can see there is a function/method, not sure what to call it that will get an array from a buffer, which I can manipulate and then get a buffer from.

My questions are...
Should I clone part of the buffer and then alter its values, is that even possible?
If I have to get an array from the buffer, would I be better going back to slice() on the array manipulating that and then getting a buffer from that?
(It is less steps that way)

I can look up the methods/functions, now Steve has pointed me in the right direction, but I don't know enough to appreciate what ids quick and lean as opposed to slow and memory hungry.

Literally any help would be appreciated.

Thanks,
Al

[Steve-Mcl]

@Dyslexicbloke

Should I clone part of the buffer and then alter its values, is that even possible?

Why do you want to do that? For what reason are you wishing to alter the values returned from PLC

Either way, Regardless of why, in the documentation I linked to, you can see writeInt, writeByte etc.

@Dyslexicbloke

all the nodes are identical but add/modify a byte offset value in the msg to allow the following node to continue reading the block of memory, or not whatever

Is this the reason you want to change values? If so, then don't! You can tack on to msg anything you want. Alternatively, set the topic of the message before returning it.

E.g. suppose you detect a value you want to process in the next node, before returning the msg out of the function, set the topic like msg.topic="stop" then in the next node you can check if msg.topic is "stop". Or you could set a bunch of variables in the msg object e.g.

msg.process = "stop pump"
msg.index = 12
return msg;

Then in the next node, you can pick these up.

[Dyslexicbloke]

No... to both
I don't anticipate an issue sending either coils or 16 Bit words, mapped to coils bit by bit, or as set-points.

Th issue is the 32 Bit data I am getting back.
Big Endian get me a valid 16 Bit value and the code you suggested works well once it has valid data but the PLC is sending 32 Bit data with the word order reversed.

Using your example I can get a float, and presumably write one, but the bytes are in the wrong order, the first and second pare, whatever their endedness, need to be swapped.

Right now the code is only viewing the array to get a value, I assume.
since I 'do not' want to alter the message returned by the PLC, to avoid the possibility of propagating an error, I am going to need a copy...
I know enough to realize that copying the entire thing in every node that uses it is just silly so getting it 4 Bytes at a time and manipulation that seems the better option.

I either have to use an array in the first place, and then get a buffer from it to do the convrsion/cast whatever
OR
Get an array from the buffer before doing the same thing as above
OR
get a partial copy of the buffer and move its elements about

The first two should be easy to find now, in fact you already told me about parts of them and there are not that many.

Overall the third option will be faster I suspect but I don't even know if it is possible.

Hope that makes more sensc

[Steve-Mcl]

Tbh, I wouldn't even be considering optimisation until I see an issue.

In this particular instance, I'd simply take the data I need, do all the conversation I need, send out multiple messages (with individual topics) - all in one function. No passing arrays or buffers around - get it done in one place.

Hint:

let v1 = msg.payload.readFloatBE(0);
let v2 = msg.payload.readInt16BE(2)
let v3 = msg.payload.readInt16LE(60)

node.send({topic:"motor 1 speed", payload: v1})
node.send({topic:"temperature", payload: v3})
node.send({topic:"light sensor", payload: v2})
return;

On the other side of that function, I'd direct the messages by topic name (use a switch node) to where they're needed.

Does that make sense?

You would do well to spend 10 minutes with the node-red documentation. Specifically "working with messages".

[Dyslexicbloke]

Sorry, I must be missing the point...

Other than the original message, which apart form a single updated offset variable, remains unchanged throughout, there are no arrays to pass about...

1 message every n seconds from a device.
extract data from it that either triggers another flow or is just displayed.

Having read the overview of Node Red and looked at example of how some simple tasks should be managed I am struggling to see where my proposed approach is not appropriate.

In your example, an I realize its just a suggestion snippet not a solution, you suggest creating a function with application specific properties.

Now I will admit I have probably spent more time watching tutorial videos and testing the concepts demonstrated than reading but it seems to me that the very heart of flow based programming is to process tasks in small chunks with standard building blocks.
in addition propagating the original throughout message, as unchanged as is practical, throughout the chunk/flow is continually recommended.

Have I got that backwards somewhere?

All previous detail aside...
My original issue, and the complication resulting from it, much of which was my lack of understanding, is sorted because you pointed me to the correct way to handle conversions.

For whatever reason the data from my device needs manipulating before conversion and I want to do that with a minimum of resource use and preferably in such a way as to make the method common throughout all my flows... Consistency, something else in many "How do I do this?" answers.

I think I get the concept but I am unsure why you are recommending a switch node approach...
you seem to be assuming that a message will be returned in response to some event hand have to appropriately handled, which of course could happen, but generally will not.

There will be messages with a specific topic... 'this device is reporting some impotent state' from some remote devices and some MQTT stuff, switches for example would do that as would remote plant that is only monitored, but generally I will be polling local devices.

Right now I have a generator and an off grid solar/battery/inverter setup.
The generator will be handled,m the fast stuff, by a local PLC whilst the solar kit is Victron

I will be adding a full suite of home automation, likely using Openhab as a quick way to build an interface although the jury is out on that so far as coding there is also required.
Most, if not all, of my distributed devices will use MQTT, probably Tasmota on Sonoff Esp8266 but that is largely unimportant right now.

I will also be adding other a solar tracker for my two arrays and likely at least 1 more.
A custom built heating system and a custom build HRV that will need entropy control

Those are just the things I know I want to do now at home.

I want to investigate a system in the wild processing M2M status updates from my customers hydroelectric and solar installations.

If I am going to start writing a new function of for every device, or set of devices, on every system why would I use Node-Red in the first place?

Ultimately, and critically for the M2M application it would be better if the system could be data driven, allowing stuff and simple rules, to be defined, based on instances of worker flows, dynamically managed by a database, but that is so far of topic right now as to be irrelevant... Basics

I am concerned I am missing something, sure I realize I have little idea about the detail but I thought I had a handle on the concepts and your comments suggest I don't.

The function I was/am working on is doing exactly what you describe exactly as you recommended doing it. Sure each node only handles 1 thing but then each node is also generic deriving its context from the index of the data it is processing, naively and without any need to customize that.

Wouldn't making task specific, essentially with hard coded context, be a huge step in the wrong direction and ultimately lead to a massive unmanageable mess?

I still want to ask about swapping my words about, not because I want to but because it bis simply required for 32 bit data but that seems to be secondary right now to understanding the way I should be planning things, generally

Regards,
Al

[Steve-Mcl]

Hi again, so you will have to bear with me here - there is of course a bigger picture - I am not trying to steer you into a final solution - your original requirements were quite far off track (splicing/Uint arrays etc) & thats ok - my responses are intended to nudge you in the right direction & into your own solutions.

@Dyslexicbloke

Sorry, I must be missing the point...

So you were concerned about passing a buffer around, asking if you should slice it, talk of adjusting values in the data for later handling etc. You went on to say things like "with a minimum of resource" and mention about "not hard coding" and "why would i use node-red" etc.

The point is, until you have a handle on node-red, most of this is all just play time. A final, roll out / repeatable / programmable / flexable single OEM type solution will take far more than a few replies on a forum thread.

for example this is graphical - but still hardcoded...

This is text in a function - and yes still hardcoded..

However, going back to your original requirement of being "efficient" and best way of doing something,
I'm pretty sure you can see in one function node, you can take all of the values you need, coerce them into the correct format (floatBE, Int16LE / whatever you need) then send them out as individual values with a topic - nicely packaged up for MQTT or whatever else you need.

the truth...

The truth is, there is not really any "best way" but there are definitely bad ways.

attempt at reply

I will now try to answer some of your many embedded questions - sorry if i miss some...

@Dyslexicbloke

in addition propagating the original throughout message, as unchanged as is practical, throughout the chunk/flow is continually recommended.

The point of not changing the message is dont return {a new message every time} but return the_original_msg; - that way you reduce cloning/duplication AND gain the ability to inspect the msg with all of its adjustments & changes during or atthe end of the flow (using debug nodes). This is definitely the way to go - normally.

My snippet that i provided you was taking a bunch of (possibly) unrelated data values from a modbus/PLC memory area, turning them into useful/correct types ans sending them on as brand new individual message - seems like I'm telling you to do the opposite now.

Well, yes and no. consider in a PLC, the memory banks will contain many many unrelated different values. However, SCADA or OPC server ideally should poll whole swathes of memory at once (thus reducing expensive COMMS) and then at server side, break them down in to useful, individual/unrelated parts. check my sample code once more that I generated (as an example) 3 unrelated messages.

I could have done this instead...

let v1 = msg.payload.readFloatBE(0);
let v2 = msg.payload.readInt16BE(2);
let v3 = msg.payload.readInt16LE(60);
msg.payload.motor1speed = v1;
msg.payload.temperature= v3;
msg.payload.lightsensor = v2;
return msg;

BUT - these three unrelated values now are all under one object & more difficult (and less useful) to send to modern platforms like MQTT (whereas the previous version I wrote was ready to send direct to an MQTT broker - albeit the topics were pretty naff - it was an example).

@Dyslexicbloke

Ultimately, and critically for the M2M application it would be better if the system could be data driven, allowing stuff and simple rules, to be defined, based on instances of worker flows, dynamically managed by a database

There is nothing to say a function cannot use a dynamic source like a database.

let dbSettings = flow.get("dbsettings"); //select top 1 settings from setup_db_table where customerId=12
let plcData = msg.payload; //a buffer 
let airFlow = plcData.readInt16LE(dbSettings.airFlowAddress);
let unrelatedSensor = plcData.readFloatLE(dbSettings.unrelatedSensorAddress); 

node.send({ topic: dbSettings.airFlowTopic, payload: airFlow });
node.send({ topic: dbSettings.unrelatedSensorTopic, payload: unrelatedSensor });

And if you want to be truely dynamic, your settings could be an array of topics/PLC addresses/PLCValueTypes - loop through them and produce the right data.

However, as spider-mans uncle said, with great flexibility comes great complex-ability

@Dyslexicbloke

The function I was/am working on is doing exactly what you describe exactly as you recommended doing it. Sure each node only handles 1 thing but then each node is also generic deriving its context from the index of the data it is processing, naively and without any need to customize that.

Why do you think you have written something generic? You have hard coded values regardless of if its a switch statement, or change/switch nodes. Unless there is a UI to allow settings change or a DB) - everything is non-generic.

@Dyslexicbloke

Wouldn't making task specific, essentially with hard coded context, be a huge step in the wrong direction and ultimately lead to a massive unmanageable mess?

Remember, i was demonstrating things based on your original struggles to convert values, not how to build a fully dynamic salable customer ready product

@Dyslexicbloke

I still want to ask about swapping my words about, not because I want to but because it bis simply required for 32 bit data but that seems to be secondary right now to understanding the way I should be planning things, generally

So you realise there are functions like readInt32LE() and readInt32BE() so I'm not certain where you are falling down here.
There is also things like swap16 if you need to byteswap everything before you start reading ints and floats. Thats the power of node/node-red.

e.g.

let buf = msg.payload;
buf.swap16();
let myFloatBE = buf.readFloatBE(0);
let myFloatLE = buf.readFloatLE(0);

[Dyslexicbloke]

@Steve-Mcl

Why do you think you have written something generic? You have hard coded values regardless of if its a switch statement, or change/switch nodes. Unless there is a UI to allow settings change or a DB) - everything is non-generic.

Ok I get that point and I don't disagree at all... Its about context / scope whatever you want to call it.

I am well aware that the more generic a lump of code is the more configuration it will take to make it do anything useful and I also know that the returns are diminishing the more generic you go.
Heck that is the whole point of Node-Red inst it?

It offers task-centric processing based on a common interface container that is shaped to an applicable form by the task in-hand.

Looking at the working with messages, again, in more detail and with better understanding now, I can see that split, change and join are more powerful than I originally realized and I also now appreciate the distinction between sending multiple messages, serially, and working with just one big one.
The point I think you were making with the two alternative structures you suggested.

But back to generic...
No I don't think I grated some thing generic in an absolute sense, of course it will only work in context, in this case processing data from one specific type of communication node and performing actions on that message.

However, using either structure you suggest would require code to be altered every time I added a a new source of data of that type or changed/augmented what was being returned,
That seems to fly in the face of the basic underlying strategy of Node-Red.
I get that I will never simply add a value to a block of data and have it magically appear on my Ui without configuring something but if the something can be a 'standard node' as opposed to custom code wouldn't that be better?
I an including a configurable function in the concept 'standard node' here.

When I say Generic, in this specific context, I am saying my function, or stack of identical functions, don't need to know what they are processing just what to do with the data, making their function generic, all be it for a specific task on a specific type of message...
Need a new value just add another identical node to get the next value.
Need to skip a few registers, insert a change node to modify the current offset, and then add another identical node.

Looking at the examples that seems entirely consistent with what is there now.

Weather I use a split change join or a branch structure is a very good point and one I need to understand more fully...
I am not aware yet of the finer points, and to be fair only just aware of the core principles, so I am in no place right now to go asserting which architecture will work best in this case.

I am very happy to conceded that my function, heck any function, may be better changing/appending the original message or splitting it into a series of new messages, on the same connector/wire,
For that matter it may be better to simply extract a value and save in global context for use elsewhere, which to be fair sounds like a good plan to me.

However at some point I need to
Identify a value based on its position in the data block.
process the raw data into a useful value, probably scaling and rounding at the same time.
give that thing a label/context to identify it.
Place the newly identified data into a database along with all previous data with similarly classified.

Mmmm... Looks like we are more or less on the same page, which is good as you will be on the right page and I have hardly skimmed the book yet.

I am assuming most of that process is easily doable with standard nodes and have no desire to go mucking about in code to do a worse job than has already been provided for by the Node Red platform.

I am also happy to concede that the bits I currently think standard nodes will not do can actually be done with standard nodes once I know how to use them.

However even the docs you pointed me to suggest a function node for tasks not already defined, and provisioned for by a standard function node.
It seems to me that jumping through hoops to split a message into multiple, potentially hundreds, of individual messages and then reassembling those into a new message, with the bytes reordered so that the standard conversion will work, would be counter productive, and hard to follow.
I would still need a node to label and format each item, or a custom function to do that for all of them, in order to put that data where it needs to be, in a useful form.

Consider...
I have a device, a PLC, that is inherently flexible, and its doing a few jobs.
I could set up multiple queries to small blocks of memory associated with each PLC task.
If I do that every time I add or modify a function on the PLC I would need to add a new getter , and setter, and a new stream to process that data. I would also have to consider the impact on existing streams that also talk to that device.
Or
I can define one block of memory to work with, handle it once per data exchange and have my PLC code use that block or all UI and integration functionality.
Going that way, a single stream as only two tasks,
Get and format data - make it available globally
Monitor global data, associated with that device - format and send the data if anything changes.

In either case I have to manually identify the data, essentially giving an offset, as the defining key, a name and format. This is no different conceptually than naming a memory address.
In Node-Red if I can do that simply by dragging a node into place and giving it a couple of set points surly that is better than having to mess about editing an existing node that is tested and working.

I build industrial applications for a living and the one thing I strive to avoid, all the time, is long gangly code that dose multiple things, mainly because editing that can be dangerous, literally life and limb stuff in many cases.

Look I take the point RE multiple outputs VS multiple messages on a single output is a thing I need to understand, the finer points of and don't. Given that node red is single threaded I suspect I need to know a great deal more to decide which is the best approach, why and when.
I am going to look at the docs and YouTube to see when a switch is recommended and what for, specifically in comparison to split type operations. That seems pertinent. is it?

With regard to building functions that can only handle a specific set of named things. as opposed to a function to be used multiple times and configured, simply, to handle one thing.
Well I an not convinced yet and the latter seem like the better plan.

Everything else I do is based on standard functions and node red is no different, configurable nodes to do common tasks.
Why would I build a function that wasn't configurable?

In all honesty what I relay want, ultimately, is a data object...
Within the object would be a name, a format for the raw data, a source, and a format for the displayed data, as a minimum.
Now I think a fully configured source for all data is a panacea too far so making the source an identifiable function block on a named stream steam seems reasonable.
Ans of course the address, offset, of the data is both unique to a block and implicit to the function.
Given that, there is no reason that said function couldn't use the data item to configure its functionality... Simply adding it would configure it, assuming the data was defined first.

Could a single function do that for multiple items of data.. Probably, but it would be way more complex.
Are there already nodes that do that... I don't know but I will find out before I go trying to make one.

If I start with the concept of one node one data item, no matter if the message is serial or parallel, I can start simple, data config in the node, and update that later if the data object idea gets implemented.

Looking where IOT is going and how that will impact industrial automation I think OPC is likely to get sidelined in favour of timeline databases and pushing value pares. without pre-configuring the underlying structure seems like the way forward for most of this stuff.
Lets face it if you need a classic SQL view for some reason and have to normalize the date tou can still do that, its just that all that pesky mucking about with com objects, and paying DB Admins to sort out the resulting mess, is long gone and the timeline tables form the input structure.

Which I guess gets us back to the original question...
Irrespective of the arrangement of the function, with respect to its handling of the original message, I still have to swap words in the underlying data.
I could process the original message and modify it, but I would rather not do that as it would ,make debugging harder and inst a recommended strategy anyway.
That mean I have to copy/clone the data, preferably a chunk at a time, 4 bytes or 2 words, depending on weather it is an array or a buffer we are talking about.

If I went with your example and built on that, which I very well might, I still have to manipulate the byte/word order for 32 Bit data and I am likely to want to cherry pick those segments from a much bigger data set.

Can I process the buffer items directly?
I will be looking at NodeJs now but pointers wouldn't hurt as there are probably many possible ways to go an only a few good ones.

If not, and I am therefore working with arrays, am I better using the array data in the first place.
I realize I have to get a buffer to do the conversion, that's fine

So...1-3 ?
(Original data) buffer > array from part of that > mess with byte order > buffer > conversion(0)
(Original data) array > array from part of that > mess with byte order > buffer > conversion(0)
(Original data) array > mess with byte order > buffer > conversion(n)

The last thing that occurs to me is that the convert function its self may be an option.
I know you can add a function to a prototype cold I access the code for the conversion, make a new version handling the bytes in a different order and then add that to the environment?
That would be clean...Potentially.
I am so far out of my depth here its hard to know if that is just a silly thing to say.

I want to move forward so I am going with the first solution, clunky or otherwise, I find for now.
I will modify once I have a better Idea what I am doing but for now getting the value, any way that works, will do,

Thanks,
Al

[Dyslexicbloke]

@Steve-Mcl

So you realise there are functions like readInt32LE() and readInt32BE() so I'm not certain where you are falling down here.
There is also things like swap16 if you need to byteswap everything before you start reading ints and floats. Thats the power of node/node-red.

Yes I got that thanks...

I don't know the actual order, and I largely don't care given that it is what it is so please, no posts telling me the byte order I ma using is wrong please, this is just an illustration.

Assume my 16 bit data is BE, because it is so...
BBAA DDCC I assume, not that it matters (low address and bits to the right as shown)
using readInt16BE(n) works just fine, it would be reasonable to assume that...
BBAA DDCC FFEE HHGG would be a BE 32Bit number
However using readInt32BE(n) readFloatBE(n) is failing until I do this...
FFEE HHGG BBAA DDCC

Honestly I couldn't care less re the academics here.
I don't think I can change the endness of the modbus-flex-getter and using the LE version of the 32 Bit reads doesn't fix it. swapping the word order works just fine.
I tested this by doing it manually in the PLC, which I also cant change the native order of by the way.

I think it is a modbus thing, I seem to remember more than one standard for data transfer updated by IIEE at some point but again it doesn't matter, it is what I have.

If I missed something I am happy to be corrected but BE/LE isn't it.

I use these PLC's, Schneider M221, daily BTW and every HMI I ever connected to one just works with standard settings in it com block.
I have no Ida why his isn't working as-is but fixing it seems like an easy task.

@Steve-Mcl

There is nothing to say a function cannot use a dynamic source like a database.

That is a huge help, thanks

Thanks
Al

[Dyslexicbloke]

That Modbus thing...
Modbus endness

I think pot luck describes it best... Ah well on and up.
It seems odd it isnt a thing in the Flex_Getter, perhaps it is and I missed it.
That said, as my PLC demonstrates, there is no guarantee the 16Bit and 32Bit will be handled the same way in fat the page I linked suggests that would be unusual.

Re Buffers swap16() and swap32() work well and are simple to implement and I have done much more reading on the entire subject, so now I am just a novice and not entirely clueless.
I think the correct term would be conscious incompetent.

One interesting point... swapXX(), and its kin, act on the entire buffer, that had me scratching my head for a while.

so doing...

MyVar2 = msg.buffer.swap32().readFloatLE(2);

Looks like a good plan but inst, despite the fact that it works.

MyVar1 = msg.buffer.readInt16BE(0)
MyVar2 = msg.buffer.swap32().readFloatLE(2); // different byte order required for float
MyVar3 = msg.buffer.readInt16BE(6)

Above, MyVar3 would be wrong because the swap32() above it has altered the data.
Run it again without getting a new set of data and only MyVar3 will work.
it is clearer to think of it like this...
(which has the same effect but clearly isn't doing what the comment suggests is the plan)

MyVar1 = msg.buffer.readInt16BE(0) 
msg.buffer.swap32();
MyVar2 = msg.buffer.readFloatLE(2);  // different byte order required for float
MyVar3 = msg.buffer.readInt16BE(6)

Of course...

MyVar1 = msg.buffer.readInt16BE(0)
msg.buffer.swap32();
MyVar2 = msg.buffer.readFloatLE(2); // different byte order required for float
msg.buffer.swap32();
MyVar3 = msg.buffer.readInt16BE(6)

will put it right again but that a load of processing on a big array, and easy to muck up
buffer.slice(), depreciated apparently, and buffer.subarray() do not copy buffers so the underlying buffer is still referenced and thus changed by swap()
(or at least it would be if it didn't thorough a RangeError for anything but the entire array)

The fix, or perhaps I should say, the method I am using now is ...

const bufRefIn = msg.payload.buffer.subarray(SrcOffset, NxtOffset);  // Get a refrence to part of the buffer
const bufLocal = Buffer.from(bufRefIn);                              // Copy that to isolate changes

// Note that the swapNN() affects all date in the buffer, thus swapping a single number is not possible                                                                     
if (SwapByteOrder){bufLocal.swap16();}                         // Swap Bytes in 16Bit Words > LE
if (SwapWordOrder){bufLocal.swap32();}                         // Swap Word16 order for 32Bit numbers > BEW

That leaves the original message alone and still permits the byte order to be manipulated, if needed.

It doesn't matter of course how the message is handled but interestingly going with a block of conversions in a single function would require the byte order to be considered and probably the indexes to be handled out of sequence to avoid repeatedly swapping bytes about.
Of course to follow the guidelines you would also want to restore the original message structure, or end the message.

My function is now working.
Weather it is the right approach? I think I need to learn more and head advice.
The whole multiple serial message thing is complex, potentially but I can see has clear advantages in some respects, specifically everything is always available wherever you ere in a flow.
(although my function still dose that on both outputs.)

I am going to look at scope storage dashboard and integration with other stuff before deciding how to proceed but I would very much welcome further comment.
Foolish to think I am anywhere near approaching correct at this point... Learning though

Thanks so much Steve,
You pushing me in the right direction has been extremely enlightening.

Al
Function code Code below...

/********************************************* Notes **************************************************************

Takes an array, 8 Bit or 16 bit values, and extracts 16 or 32 bit values.
Processing starts at 'Offset' in 'msg.payload.buffer'
The output is a single value in 'payload.value' on output 1
msg is passed to the remaining flow, output 2, with Offset set for the next value

Byte / word order see ...
https://ctlsys.com/support/common_modbus_protocol_misconceptions/

********************************************** Setup **************************************************************/
var OutType = 1;                                               // 1-INT16 2-Int32 3-Float32
var SwapByteOrder = 0;                                         // SBO - Little Endian 16 Bit Data, Byte Order
var SwapWordOrder = 0;                                         // SWO - Big Endian 32 Bit, Byte and Word order
                                                               // SBO & SWO can be used in conjunction
/*******************************************************************************************************************/

var SrcOffset = msg.offset || 0;                               // Get the current offset = assign 0 if undefined
var ConvRslt = 0;
var NxtOffset = 0;
var strStat = " @ " + SrcOffset.toString();
                                                              
var msgResult = RED.util.cloneMessage(msg);                    // Clone msg Object, for the result on output 1

msgResult.payload = { value:null }                             // Clear the origional payload leaving the rest intact
                                                               // Init with just the one item!
switch(OutType)
{
  case 1:                                                     
    strStat = "Bytes to Int16 " + strStat + " > ";             // Set lengt and status text asociated with output
    NxtOffset = SrcOffset + 2;                                 // Set Next Offset from offset and byte length
  break;
  case 2:
    strStat = "Bytes to Int32 " + strStat + " > ";
        NxtOffset = SrcOffset + 4;
  break;
    case 3: 
    strStat = "Bytes to Float32 " + strStat + " > ";
    NxtOffset = SrcOffset + 4;
  break;
}                                                              
                                                              
const bufRefIn = msg.payload.buffer.subarray(SrcOffset, NxtOffset);  // Get a refrence to part of the buffer
const bufLocal = Buffer.from(bufRefIn);                              // Copy that to isolate changes

// Note that the swapNN() affects all date in the buffer, thus swapping a single number is not possible                                                                     
if (SwapByteOrder){bufLocal.swap16();}                         // Swap Bytes in 16Bit Words > LE
if (SwapWordOrder){bufLocal.swap32();}                         // Swap Word16 order for 32Bit numbers > BEW 

switch(OutType)                                                // NB. swap16() makes BE LE, no need to call readxxxXXLE
{
  case 1:                                                            
    ConvRslt = bufLocal.readInt16BE(0);                        // View the local buffer as Int16, get its value
    strStat = strStat + ConvRslt.toString();                   // Add the value to the status string
  break;
  case 2:
    ConvRslt = bufLocal.readInt32BE(0);                        // View the local buffer as Int32, get its value
    strStat = strStat + ConvRslt.toString();
  break;
    case 3: 
    ConvRslt = bufLocal.readFloatLE(0);                        // View the local buffer as Float32, get its value
    strStat = strStat + ConvRslt.toString();                   // LE here refers to both Byte and Word Order
  break;
}

node.status({text:strStat});

msg.offset = NxtOffset;                                        // Set the position to start the next value extract process.
                                                               // This will add the property if it dosnt exist
msgResult.payload.value = ConvRslt;                                                           
return [msgResult, msg]                                        // TOutputs, 2 has an updated offset value, 1 is the result.

[Steve-Mcl]

Good morning @Dyslexicbloke

@Dyslexicbloke

I don't know the actual order, and I largely don't care given that it is what it is so please, no posts telling me the byte order I ma using is wrong please, this is just an illustration.

I don't believe ever told you your byte order was wrong? Did I?
I am only trying to help you help yourself and learn something in the process.

So, this (below) is why I addressed the 32bit subject (you asked)...

@Dyslexicbloke

I still want to ask about swapping my words about, not because I want to but because it bis simply required for 32 bit data

Tbh, as much as we try to convey our requirements or intent, text alone is never enough to get everyone on the same page.

For example, I didn't fully digest your early posts (they were incredibly long) and you didn't supply every piece of context or the inner working of your brain or the final goal of this particular thread.

Its just not possible.

The point i'm trying to make is, if my initial replies were short / borderline curt it's only because they were simple steers.

---

Back to the 32 BE/LE thing...

Yes, I suspect you are missing the point here.
As I know you realise, data is transferred in varying formats and has to be reordered either at source (in the PLC) or at destination. It truly doesn't matter where. So my point is do it at destination since it is far easier & 100% possible to do in node red.
Please keep reading...

@Dyslexicbloke

I don't think I can change the endness of the modbus-flex-getter and using the LE version of the 32 Bit reads doesn't fix it. swapping the word order works just fine.

I'm not sure you can change endian-ness in the modbus nodes - but you can within node-red - what I was attempting to steer you to.

Let me demonstrate...

@Dyslexicbloke

Assume my 16 bit data is BE, because it is so...
BBAA DDCC I assume, not that it matters (low address and bits to the right as shown)
using readInt16BE(n) works just fine, it would be reasonable to assume that...
BBAA DDCC FFEE HHGG would be a BE 32Bit number
However using readInt32BE(n) readFloatBE(n) is failing until I do this...
FFEE HHGG BBAA DDCC

So I will use your sample data...

BBAA DDCC FFEE HHGG
should be
FFEE HHGG BBAA DDCC

paraphrased (for use in a dynamic example)...
2211443366558877
should be
6655887722114433

//original data example  2211443366558877
//wanted data format     6655887722114433
buf.swap64()
//after buf.swap64       7788556633441122
buf.swap32()
//after buf.swap32       6655887722114433

At this point, the buffers format is as you hoped & thus the readXxxx32YY functions will operate as expected.
Working example

---

I will read through the rest of your posts shortly (you may have have possibly resolved this by now).

Regards, Steve.