OPCUA data parsing issue

I'm having some difficulty understanding how to parse certain values from the data returned from the opcua client.

The data that is being subscribed to looks like this:

{e[36m /*RunInfo*/e[39m
e[33m jobGuid                      e[39m e[36m/* Guid                */e[39m: 00000000-0000-0000-0000-000000000000
e[33m planGuid                     e[39m e[36m/* Guid                */e[39m: 00000000-0000-0000-0000-000000000000
e[33m runGuid                      e[39m e[36m/* Guid                */e[39m: 00000000-0000-0000-0000-000000000000
e[33m sortGuid                     e[39m e[36m/* Guid                */e[39m: 00000000-0000-0000-0000-000000000000
e[33m runNumber                    e[39m e[36m/* Int32               */e[39m: 0
e[33m cutState                     e[39m e[36m/* RunCutState         */e[39m: RunCutState.None ( 0)
e[33m cutStartTime                 e[39m e[36m/* DateTime            */e[39m: 1601-01-01T00:00:00.000Z
e[33m cutEndTime                   e[39m e[36m/* DateTime            */e[39m: 1601-01-01T00:00:00.000Z
e[33m sortState                    e[39m e[36m/* RunSortState        */e[39m: RunSortState.None ( 0)
e[33m sortStartTime                e[39m e[36m/* DateTime            */e[39m: 1601-01-01T00:00:00.000Z
e[33m sortEndTime                  e[39m e[36m/* DateTime            */e[39m: 1601-01-01T00:00:00.000Z
e[33m actualCutTime                e[39m e[36m/* Double              */e[39m: 0
e[33m actualStopTime               e[39m e[36m/* Double              */e[39m: 0
e[33m actualWaitTime               e[39m e[36m/* Double              */e[39m: 0
e[33m sheetOffsetX                 e[39m e[36m/* Double              */e[39m: 0
e[33m sheetOffsetY                 e[39m e[36m/* Double              */e[39m: 0
e[33m sheetAngle                   e[39m e[36m/* Double              */e[39m: 0
e[33m chargeInfo                   e[39m e[36m/* UAString            */e[39m: e[34mnulle[39m
e[33m storageInfo1                 e[39m e[36m/* UAString            */e[39m: e[34mnulle[39m
e[33m storageInfo2                 e[39m e[36m/* UAString            */e[39m: e[34mnulle[39m
e[33m storageInfo3                 e[39m e[36m/* UAString            */e[39m: e[34mnulle[39m
};

I'd like to display the "cutStartTime" on a dashboard and graph the "cutState"

I really don't know what to do to parse this out. Any help would be greatly appreciated!

Hi and welcome :slight_smile:

My guess is that data is intended for a terminal display (those numbers look like colour codes or formatting markers)

I would recommend you browse the OPC Server (using something like uaexpert) to see if there are any individual tags with the value(s) you desire before you start trying to parse that data.

Come back and let me know if there are no more tags available & I will help you parse it.

As @Steve-Mcl pointed out, you're picking a whole structure, not just a value.

I've been working with OPC-UA these last few weeks, if you use UAExpert as Steve suggests, you can look for the actual value you need. If you add the values to the data access view in UAExpert (drag and drop) you can make sure the value you will get is the one you need. For example, in this instance:

I should pick

ns=2;s=Machine.M01LE01MQ01#0_X_I.Value

If I want to get the numerical value. If I try to add the same minus the ".Value", UAExpert will give the following warning:
image

Which looks exactly like what you're getting there.
If you just need one value, please go into the address space, select the value you need, and just request that. You will still get a small object with the timestamps and some other info, but it is much simpler to parse.
If you request several values and use multiple reading, you will get an array of objects with the values inside them.

So in UAExpert I open the Object then select the value. What I am looking for is a "sub" value
I'd like to extract the Name from this set of values

Is this possible or do I need to write a function to parse the text?

On the left side bar, take CurrentJob, drag and drop it in the Data acces view.
Do you get a single variable, or do you get an object?

It might be that you need to add the tag

ns=2;s="Work.CurrentJob.Value.Value.Name

if you only want that value.

It looks like it is an ExtensionObject

Right, try:

ns=2;s=Work.CurrentJob.Name

Otherwise, when you get the object, you just need to call the object.Name property in node-red to get the string

EDIT: I had some random quotes that appeared before Work. That would invalidate the result.

I get null values returned when I try that.

How do I call the object.Name property do I put two OPCUA objects in series?

  1. Try adding:
ns=2;s=Work.CurrentJob

Add a debug node at the client exit that shows the complete msg object, and expand the properties. Paste the result.

This is what that looks like:

10/12/2021, 08:54:32node: cd42115dbdd8e013
ns=2;s=Work.CurrentJob : msg.payload : JobInfo
JobInfo
{e[36m /JobInfo/e[39m
e[33m jobGuid e[39m e[36m/* Guid /e[39m: 5C7DF28B-AC4F-4118-8711-5A11925CAD39
e[33m externalJobGuid e[39m e[36m/
Guid /e[39m: 759F0CBA-313A-434E-86C8-569A3AF202FA
e[33m name e[39m e[36m/
UAString /e[39m: 496_1818602
e[33m description e[39m e[36m/
UAString /e[39m:
e[33m userInfo1 e[39m e[36m/
UAString /e[39m:
e[33m userInfo2 e[39m e[36m/
UAString /e[39m:
e[33m userInfo3 e[39m e[36m/
UAString /e[39m:
e[33m measurementSystem e[39m e[36m/
MeasurementSystem /e[39m: MeasurementSystem.Metric ( 0)
e[33m ncProgramFile e[39m e[36m/
UAString */e[39m: \10.6.3.19\bysoft\lcc\496_1818602.bvc
};

Sorry for the late reply, last week was extremely busy.

That payload looks EXTREMELY weird. I think your OPC-UA server is has a problem with character coding.

First thing would be to check which character coding the OPC-UA server is using (probably not UTF-8). It should match the one in node-red. If you can change it so they match, that might solve the issue.

The sequences "e[33m", "e[36m" , and "e[39m" look to me like they stand for certain characters that, if replaced, would get you a nice, parseable JSON. You can try loading the payload as a string and removing all those character sequences first, and see then if you can use a JSON node to parse it and get the object formatted properly.

they look like terminal colour codes - eg see Jafrog's dev blog - no idea if there is an option to turn them off - or if you would have to strip them out with a clever regex or similar.

Maybe force-loading the payload object as a string, then edit them out with the string node, then parse the resulting string into JSON?

Thanks for all the help thus far!

The OPC Server is running on a windows 10 system. Its part of a bunch of other proprietary software and I'm really not sure how to tell if it is set to UTF-8 or not. My node-red system is running on arch-linux and is set to UTF-8

Thanks for the tip to force it into a string. I downloaded and installed that module and it might just be the way to do this. That said I have been able to get some interesting results Somethimes I get output that looks like this (It says Type not printable??) but the data looks more or less clean.

20/12/2021, 15:59:49node: b5117cb2a94c0f74 ns=2;s=Work.CurrentPlan : msg.payload : PlanInfo PlanInfo [Type not printable]PlanInfo { jobGuid: '82FD37E5-AB23-430E-99BA-4F477A97C2C4', planGuid: '175BE2CA-92DF-4A22-9A86-944A51D9E26D', name: '512_1849747', description: '', sizeX: 1848.515733196966, sizeY: 694.8310772215672, totalRuns: 16, totalParts: 1, planState: 3, estimatedCutTime: 3136.144, materialFormatType: 0, materialName: 'MS_XLTool', materialSizeX: 1870, materialSizeY: 695, materialThickness: 0.91, tubeProfileType: 0, profileDimA: 0, profileDimB: 0, profileDimC: 0, weight: 9.295876590000002, waste: 0.44893076028116086, articleInfo: 'GREEN 120488 XL Tool Inc.', chargeInfo: '120488', materialInfo1: '0.04 40x40"', materialInfo2: '', materialInfo3: '', parameterFile: 'BySmartFiber_L802_10000_MS_1_N2.par', spacerPlateInfo: '' }

Other times it looks like:

20/12/2021, 16:04:39node: b5117cb2a94c0f74 ns=2;s=Work.CurrentPlan : msg.payload : PlanInfo PlanInfo {e[36m /*PlanInfo*/e[39m e[33m jobGuid e[39m e[36m/* Guid */e[39m: 82FD37E5-AB23-430E-99BA-4F477A97C2C4 e[33m planGuid e[39m e[36m/* Guid */e[39m: 175BE2CA-92DF-4A22-9A86-944A51D9E26D e[33m name e[39m e[36m/* UAString */e[39m: 512_1849747 e[33m description e[39m e[36m/* UAString */e[39m: e[33m sizeX e[39m e[36m/* Double */e[39m: 1848.515733196966 e[33m sizeY e[39m e[36m/* Double */e[39m: 694.8310772215672 e[33m totalRuns e[39m e[36m/* Int32 */e[39m: 16 e[33m totalParts e[39m e[36m/* Int32 */e[39m: 1 e[33m planState e[39m e[36m/* PlanState */e[39m: PlanState.Started ( 1) e[33m estimatedCutTime e[39m e[36m/* Double */e[39m: 3136.144 e[33m materialFormatType e[39m e[36m/* MaterialFormatTyp */e[39m: MaterialFormatType.Sheet ( 0) e[33m materialName e[39m e[36m/* UAString */e[39m: MS_XLTool e[33m materialSizeX e[39m e[36m/* Double */e[39m: 1870 e[33m materialSizeY e[39m e[36m/* Double */e[39m: 695 e[33m materialThickness e[39m e[36m/* Double */e[39m: 0.91 e[33m tubeProfileType e[39m e[36m/* TubeProfileType */e[39m: TubeProfileType.None ( 0) e[33m profileDimA e[39m e[36m/* Double */e[39m: 0 e[33m profileDimB e[39m e[36m/* Double */e[39m: 0 e[33m profileDimC e[39m e[36m/* Double */e[39m: 0 e[33m weight e[39m e[36m/* Double */e[39m: 9.295876590000002 e[33m waste e[39m e[36m/* Double */e[39m: 0.44893076028116086 e[33m articleInfo e[39m e[36m/* UAString */e[39m: GREEN 120488 XL Tool Inc. e[33m chargeInfo e[39m e[36m/* UAString */e[39m: 120488 e[33m materialInfo1 e[39m e[36m/* UAString */e[39m: 0.04 40x40" e[33m materialInfo2 e[39m e[36m/* UAString */e[39m: e[33m materialInfo3 e[39m e[36m/* UAString */e[39m: e[33m parameterFile e[39m e[36m/* UAString */e[39m: BySmartFiber_L802_10000_MS_1_N2.par e[33m spacerPlateInfo e[39m e[36m/* UAString */e[39m: };

With the string node I can remove the crap but I'm not sure how to remove the unnecessary white spaces.

I end up with this: --EDIT --- This site seems to remove all the extra spaces... There are a bunch in between each word.

{e[36m /*PlanInfo jobGuid Guid : 82FD37E5-AB23-430E-99BA-4F477A97C2C4 planGuid Guid : 175BE2CA-92DF-4A22-9A86-944A51D9E26D name UAString : 512_1849747 description UAString : sizeX Double : 1848.515733196966 sizeY Double : 694.8310772215672 totalRuns Int32 : 16 totalParts Int32 : 1 planState PlanState : PlanState.Started ( 1) estimatedCutTime Double : 3136.144 materialFormatType MaterialFormatTyp : MaterialFormatType.Sheet ( 0) materialName UAString : MS_XLTool materialSizeX Double : 1870 materialSizeY Double : 695 materialThickness Double : 0.91 tub...

Is there something I can look at in UAExpert that will tell me what the server character encoding is?

Interesting problem -- Dave is correct that those are embedded terminal (probably vt100 or xterm) color codes. If this output is from some command line program or shell script, there is a chance that setting the right TERM env variable to the right value (whatever that is?) might clean up the output and return plain text, but...

I would tend to go the other way, and use those markers to parse the text using a regex. Using this JSONata regex on this try.jsonata.org sample data:

payload
    .$split(/\n/)
    .$match(/^e\[33m (\w+) .*: (\S+)/)
    .[groups] {
        $[0]: $[1].$replace(/e\[3[49]m/, '')
    }

you can see the output is valid JSON, ready for use in any downstream graph nodes:

{
  "jobGuid": "00000000-0000-0000-0000-000000000000",
  "planGuid": "00000000-0000-0000-0000-000000000000",
  "runGuid": "00000000-0000-0000-0000-000000000000",
  "sortGuid": "00000000-0000-0000-0000-000000000000",
  "runNumber": "0",
  "cutState": "RunCutState.None",
  "cutStartTime": "1601-01-01T00:00:00.000Z",
  "cutEndTime": "1601-01-01T00:00:00.000Z",
  "sortState": "RunSortState.None",
  "sortStartTime": "1601-01-01T00:00:00.000Z",
  "sortEndTime": "1601-01-01T00:00:00.000Z",
  "actualCutTime": "0",
  "actualStopTime": "0",
  "actualWaitTime": "0",
  "sheetOffsetX": "0",
  "sheetOffsetY": "0",
  "sheetAngle": "0",
  "chargeInfo": "null",
  "storageInfo1": "null",
  "storageInfo2": "null",
  "storageInfo3": "null"
}

I'm using a fairly loose pattern matcher, which only expects a single : between the fields and values, and no white space characters in the values. So you may want to tweak the regex if you find missing value data, and you may want to add some data type coersion to numbers, and to return actual nulls instead of the "null" strings.

3 Likes

This makes much more sense now.

I'm not familiar with regex voodoo, so I would probably have had more problems parsing the whole thing. It might have worked (eventually), but your solution seems more elegant.

That's sweet. I really appreciate that.
What node do i put the JSONata code into? Do I put this in a switch node? I'm very new to node red, please excuse my ignorance. :slight_smile:

Use a change node, and select the JSONata pulldown (the J:) option
Then click the three dots next to the input field to open the expression editor/tester
Paste the expression into the top window,
replace the "hello world" string with your data into the left Example window,
and your results will appear magically in the right Result window.

Ok. So I added a change node then did the following:

Add the regex script

After deploying I get the error:
Invalid JSONata expression: Context value is not a compatible type with argument 1 of function "split"

Here is a screenshot. I think there are other characters there causing grief?

I think you should copy the full msg into the Test window. Make sure you set the msg objekt to full and then copy the the whole msg objekt and paste that into the Test window.