Requiring a msg structure - enforcing the presence of fields and attributes

General
1

Hi There!

I'm sure this question has been posed many times before but I didn't find anything quickly - therefore: sorry for the repetition!

The core of the question is: is there a way to define the structure of a message and have a node check/enforce that structure?

Example is this flow which expects certain fields to be defined on the msg object. To describe that, I have a code snippet but no node in the flow that enforces that requirement:

msg.email = {
    /* addresses */
    from: "sender@example.org",
    from_name: "Example Robot",
    to: "recipient@example.org",
    cc: false,
    bcc: false,

    [... rest ignored ...]
}

I'm not a particular fan of typed systems but for pre-defined flows it would be great to have a clear interface for flow usage.

This question is less about typing (integer,string,float) and more about structure ("object requires field XYZ"). In the above example, I'm less worried whether the field 'cc' is boolean or string, rather that the field 'cc' is defined on the attribute email on the object msg.

Cheers!

2

If you use a switch you can set the test to the `Property' to the part you are looking for and then check 'is not null' and if it exists it will be true otherwise it s false. Here is a sample flow:

[{"id":"5f9754f292889ff1","type":"function","z":"be8d973fa360b639","name":"function 1","func":"msg.email = {\n    /* addresses */\n    from: \"sender@example.org\",\n    from_name: \"Example Robot\",\n    to: \"recipient@example.org\",\n    cc: false,\n    bcc: false,\n}\nreturn msg","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":320,"y":160,"wires":[["24d30be3b19938c9","1ba1a58c17708352"]]},{"id":"7e67b700f1909d6e","type":"inject","z":"be8d973fa360b639","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":140,"y":160,"wires":[["5f9754f292889ff1"]]},{"id":"24d30be3b19938c9","type":"switch","z":"be8d973fa360b639","name":"check for msg.email.from_name","property":"email.from_name","propertyType":"msg","rules":[{"t":"nnull"},{"t":"else"}],"checkall":"true","repair":false,"outputs":2,"x":550,"y":100,"wires":[["5b595dc12c7b2387"],["ce50aabefa77d7a5"]]},{"id":"5b595dc12c7b2387","type":"debug","z":"be8d973fa360b639","name":"exists1","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":780,"y":80,"wires":[]},{"id":"ce50aabefa77d7a5","type":"debug","z":"be8d973fa360b639","name":"not exists1","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":790,"y":120,"wires":[]},{"id":"1ba1a58c17708352","type":"switch","z":"be8d973fa360b639","name":"check for msg.email.meal","property":"email.meal","propertyType":"msg","rules":[{"t":"nnull"},{"t":"else"}],"checkall":"true","repair":false,"outputs":2,"x":550,"y":200,"wires":[["dde6840fc2ca6993"],["68da4150f0f773e4"]]},{"id":"dde6840fc2ca6993","type":"debug","z":"be8d973fa360b639","name":"exists2","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":780,"y":180,"wires":[]},{"id":"68da4150f0f773e4","type":"debug","z":"be8d973fa360b639","name":"not exists2","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":790,"y":220,"wires":[]}]
3

Good idea but that's a switch per property plus the overall structure does not become clear because there are so many individual switch. I admit I have no better solution and a series of switches would definitely do the job.

Of course I could also just generate those switches based on a structure that I define .... evil laugh.oO(dynamically generating a working flow in the frontend ... it does feel like UML and the early 2000s)

4

If you want to get complicated, you could define a class MyEmailClass that has those values set in the constructor where you would do the validation on the values to ensure they are set. Later, you could check that your object instance is myEmailObj instanceof MyEmailClass. The class would have to be accessible on wherever you create an instance and then later check the instanceof. That should be able to be set in the global context, but I have not tried.

1 Like
5

Wait what?!?! :slight_smile: I can define classes in Node-RED? OMGosh, this is really getting very UML'isk - I remember generating Java classes from UML specifications - mind blowing stuff at the time but a real pain to code against - infinitely long class names resulted with infinitely long method names ...

But yes, what I am talking about are definitely Java interfaces 2.1!

6

Just for info
You could use JSON schema in the JSON node to validate properties and values.

7

Javascript supports classes so yes, you can define classes in a function node.

8

I recently used the swagger nodes they do a really good job in defining an API interface. Perhaps there is a way to use them to make a formal definition for the requirements of a flow...

9

AsyncAPI (https://www.asyncapi.com) may be a better match than OpenAPI (Swagger)
and as already mentioned there are some JSON schema checking nodes in the flows library also.

10

I just found json-multi-schema nodes - I was going to give them a whirl on the FJ....

11

A basic validator is also built into the core JSON node -

image
image1026Ă—476 40.1 KB

1 Like
12

Noooooooooo :frowning: I just created a flow using the full-msg-json-schema-validation node that does exactly what I wanted - doh!

But I'll update that presently with these new details! Cheers :+1:

13

You should know by now, you should always read the node help text. (some witty smiley that i can never bothered to use)

14

RTFM? Hm I always wondered what that meant ....

In my defensive, I was not thinking of converting my payload to JSON (using a JSON node) so that a JSON node (but another JSON node) would parse it with a JSON schema (the JSON schema being in a template node) to tell me that the original payload was ok! :wink: That is a little too creative even for me!

15

In your defense, But you are industrious and get there in the end.

16

most of the time, the end is just the beginning of something else ... ends seem to be blind corners.

but in the end, it's the journey not arrival that counts.

2 Likes
17

Turns out that using the JSON node is just as good even if there is an extra conversion step - updated flow.

Definitely solves my intended use-case and will make, imho, sharing code simpler since its possible to define clear requirements on the msg object.

1 Like
18

Hmm. Maybe there is an enhancement to the json node required to add a mode where it just checks schema and doesn’t convert ?

1 Like
19

Isn't that a separate validation node? It might be confusing to add that functionality to the JSON node - I dunno.

I assume it would be a fourth option here:

Screen Shot 2023-09-20 at 10.19.43

Something like "Validate and maintain format"

Since the node does validation by default, emphasis should be placed on maintaining the format, i.e. "Validate only" would be confusing since it does not convey the purpose of the option.

EDIT:

If the JSON node does have a "validate only mode" then it should be possible to include the schema in the node itself, like the msg-object-...tion node above. That way the JSON node can act as a gateway to the usage of a flow and it would only be one node (instead having a template node containing the schema and passing it on to the json node).

20

Just for those that come after me: I ended rolling my own validator function node because I wanted to check attributes on the msg object. This isn't possible with the either the JSON node nor the msg-validator node because they can only validate a property of the msg object.

So my request would be that if the JSON node is extended as a validator, then please allow validation on the top-level msg object and not just on a property of the msg.

This is a simple example of a schema and the home-grown function node:

[{"id":"9d516b0b284e8481","type":"template","z":"36690f145d5af6ca","name":"schema","field":"schema","fieldType":"msg","format":"json","syntax":"plain","template":"{\n    \"title\": \"Email data required on the msg object\",\n    \"type\": \"object\",\n    \"required\": [\n        \"email\"\n    ],\n    \"properties\": {\n        \"email\": {\n            \"type\": \"object\",\n            \"required\": [\n                \"to\",\n                \"from\",\n                \"subject\"\n            ],\n            \"properties\": {\n                \"from\": {\n                    \"type\": \"string\",\n                    \"description\": \"Sender email.\"\n                },\n                \"from_name\": {\n                    \"type\": \"string\",\n                    \"description\": \"Senders full name.\"\n                },\n                \"to\": {\n                    \"type\": \"string\",\n                    \"description\": \"Receiver email.\"\n                },\n                \"cc\": {\n                    \"type\": \"string\",\n                    \"description\": \"CC email or blank.\"\n                },\n                \"bcc\": {\n                    \"type\": \"string\",\n                    \"description\": \"BCC email or blank.\"\n                },\n                \"subject\": {\n                    \"type\": \"string\",\n                    \"description\": \"Subject Line of email\"\n                }\n            }\n        }\n    }\n}","output":"json","x":372,"y":1091,"wires":[["9efdc1c898e037ec","62a6aee706773c33"]]},{"id":"a14424b157213fb4","type":"function","z":"36690f145d5af6ca","name":"JSON Schema validator","func":"var validator = new Ajv({\n    allErrors: true,\n    messages: true\n})\n\nconst validate = validator.compile(msg.schema)\n\nconst v = validate(msg);\n\nif ( !v ) {\n    msg.errors = validate.errors;\n    node.error(\"validation failed\", msg)\n} else {\n    delete msg.schema;\n    return msg;\n}","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[{"var":"Ajv","module":"ajv"}],"x":774,"y":1091,"wires":[["88dad74cd25d507d"]],"outputLabels":["ok"]},{"id":"9efdc1c898e037ec","type":"function","z":"36690f145d5af6ca","name":"valid data","func":"msg.email = {\n    cc: \"someemail\",\n    to: \"asdsa\",\n    from: \"dddd\",\n    subject: \"hello world\"\n}\n\nreturn msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":544,"y":1091,"wires":[["a14424b157213fb4"]]},{"id":"7eb33583b79df7f6","type":"inject","z":"36690f145d5af6ca","name":"Trigger","props":[],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":187,"y":1091,"wires":[["9d516b0b284e8481"]]},{"id":"88dad74cd25d507d","type":"debug","z":"36690f145d5af6ca","name":"msg is good","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"email","targetType":"msg","statusVal":"","statusType":"auto","x":1063,"y":1092,"wires":[]},{"id":"036e12f9db5b7590","type":"catch","z":"36690f145d5af6ca","name":"","scope":["a14424b157213fb4"],"uncaught":false,"x":803,"y":1124,"wires":[["256142c98e44b3d0"]]},{"id":"256142c98e44b3d0","type":"debug","z":"36690f145d5af6ca","name":"msg is bad","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"errors","targetType":"msg","statusVal":"","statusType":"auto","x":1064,"y":1125,"wires":[]},{"id":"62a6aee706773c33","type":"function","z":"36690f145d5af6ca","name":"invalid data","func":"msg.email = {\n    cc: \"someemail\",\n    to: \"asdsa\",\n}\n\nreturn msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":557,"y":1148,"wires":[["a14424b157213fb4"]]}]