Asking about automatic message *parsing* / construction

[Trying_to_learn]

(Sorry if this isn't exactly the right place, but I think it is just a bit more than general discussion and could be a developmental/Feature request thing.)
In light of recent events for me with MQTT, this is a bit of out loud thinking:

I get that NR with its message and MQTT are two different animals and speak different languages (if you will).

I have just been through a bit of a painful ordeal (well, ok: I guess I deserved it.) with sending messages over MQTT.

Thought:
A node that will look at an incoming message and construct a MQTT compatible message (and/or) deconstruct (why is spell check not liking that word?) an incoming message and break it back to Node-Red format?

Yes, most people wouldn't want/need it. But, now and then you are needing to send a more complex message than simply msg.payload.

When you have such a case, you could use this node at both ends and save a lot of messing around.

Example:
At the sending end of the message.

msg.payload = "Something important"
msg.device = "The device's name"
msg.*something_else* = "Other information needed"
*and so on*

(and I'm not sure what happened there with formatting)

Can NR see there are other items than the usual extra parts of the msg and build a MQTT compatible payload?

like:
msg.payload.payload = msg.payload, msg.payload.device = msg.device, msg.payload.something_else = msg.something_else

And then at the other end extract that structure to its original?

I'd love to give it a go, but I don't think I have enough knowledge / confidence yet to do it.

But I am discussing it for consideration.

[knolleary]

I think you are suggesting a node that automatically copies all top-level properties (msg.device, msg.*something_else* ) under msg.payload.

That means you have to make sure the message only contains properties you want to publish. For example, you'd get the _msgid property published as well, plus anything else that happened to still be there. In that case, you'd end up having to use a Change node to delete the things you didn't want to publish.

If you are having to use a Change node to tidy up the message before passing to this new node anyway, you may as well cut out the middle-man and use the Change node to get msg.payload into exactly the format you want.

[Trying_to_learn]

Thanks Nic.

Yes, I do know of these parts, but am not sure if they are/could/would be helpful for people with more knowledge than me to use for debugging if there are problems.

Given this node would be used either in to or out of a MQTT node, it would have to be a given that the message (in the bigger picture) only had things wanting to be sent.

Yes, I get that it would mean (possibly) a clean up of other things not wanted.
But if the flow is written correctly (?) the message would be constructed by things from a clean slate.
So - all things being good - there shouldn't be any extraneous parts.
If there are, well they would need cleaning.

And yes I agree with the last paragraph. It was only me thinking aloud.
But on the other end of the MQTT trail, a node which automagically de-constructs the message back to Node-Red format would be nice.

Yes?

[knolleary]

@Trying_to_learn

message back to Node-Red format would be nice.

The message is already in "Node-RED format".

You can access all of the properties as msg.payload.whatever. That's a given. If you build a flow that expects the value to be somewhere else, you can use a Change node to move it to where you want it to be.

[Trying_to_learn]

Yeah, ok.

I was more thinking about making the MQTT pipe transparent to the message.

So it goes in with (eg:)

msg.payload = "foo", msg.topic = "Topic", msg.device = "Devicename" (and so on)

and at the other end of the MQTT tunnel, it comes out the same. (Of course with the extra node on the receiving end.)

Again: It is just a thought.

Or! And I just thought of it now......

Although there is the a parsed JSON object option, add something else to do what I was suggesting in that node.

But that may be stepping on MQTT's toes. I don't know where the demarcation line is for what you can do in what node.

[cymplecy]

So I'm translating your idea as to being an "MQTT link node" to go between different Node-RED instances?

[Trying_to_learn]

Well, (after my little fiasco) where I forgot the MQTT tricks, and thinking at a different level:

A machine/flow/what ever has a message which needs to be sent over MQTT.
It has some (I can't remember the terms - sorry) other sub parts which are part of the message.

Ok, I have to construct the message with a change node (or function) to put it all in the msg.payload part. Then send it over/via MQTT.

At the other end it would be nice/handy to either have the node go that extra step (as I only just suggested) and put it all back to the original format, or have an extra node which de-constructs the msg.payload.(blah) back to what it was before going into MQTT.

So sorry if that doesn't answer your question clearly. But I think I have.

(P.S. After a few more brain cycles: Yes. It would be like a MQTT link node.)

But be it that is an extra feature added to the MQTT in node, or a whole new node.....

[TotallyInformation]

You would need some kind of special flag in the msg content to indicate to the -in node that it needed to do something different. Otherwise it would be a breaking change for the mqtt nodes.

This is, in fact, very much an edge-case I think. The reason I think that is that MQTT is a lightweight msg format and most brokers have limits on the msg size they allow as do all of the clients. So sending serialised JSON to MQTT is itself an edge-case, just a very common one! The truth is that the content of an MQTT topic is a string, not a structure. In my view, if you send it anything other than a simple string, you should be thinking about how you are processing it anyway.

As such, I would use a subflow or a pair of subflows to achieve this without any need to change the mqtt node.

[cymplecy]

@Trying_to_learn

Yes. It would be like a MQTT link node

@Trying_to_learn

But be it that is an extra feature added to the MQTT in node, or a whole new node.....

I think it would be best to have one node that packages the Node-RED message parts and feeds that into a standard MQTT out node

And then have another node on the output of the receiving MQTT in node to decode it back into its original Node-RED message format

[Colin]

I have never that I can remember had to do this, I wonder whether you might like to consider changing how you write this data to mqtt. Looking at your previous thread you appear to have msg.device with presumably some device identifier there, msg.status, with presumably the status and msg.payload with something else, and you are sending this to a topic which, again presumably, is used for multiple devices. Instead you could have topics like
something/deviceid/status to send the status to and
something/deviceid/data or similar, to send whatever is in your payload. Then, for example you would for example be able to subscribe to all the data for a particular device by subscribing to something/deviceid/#.

[Trying_to_learn]

Colin,

I do agree with you in some of things.

The honest truth is that a lot of this is/was written YEARS ago when I was just starting with Node-Red and I was a bit trigger happy with old habits of doing things the old way, rather than the new way.

That was only an example and just now I am up to my arms with updating a few machines with newer (better?) code, and the heart aches it is giving me.

I shall have to look at the structure of the MQTT messages: yes.
I think there is some duplication of information being sent.

Alas just now, I've been awake for ..... 16 hours, there about. I think I will have to hit the sack soon/shortly.

[cymplecy]

I've had a bit of a play and this seems to to work

This code in a function node should be placed before an MQTT out node

It moves all the msg objects into msg.mqttPackagedMessage
It removes the _msgid as that will be recreated by Node-RED when it is decoded at the receiving end

var mqttPackagedMessage;

mqttPackagedMessage = ({ "payload": msg });
delete mqttPackagedMessage['payload']['_msgid'];
node.send([mqttPackagedMessage]);

Then this code should be placed in a function node after the receiving MQTT in node

var mqttPackagedMessage;

mqttPackagedMessage = (JSON.parse((msg['payload'])));
node.send([mqttPackagedMessage]);

[Colin]

@cymplecy

mqttPackagedMessage = (JSON.parse((msg['payload'])));

Is that line not required if Parsed JSON is selected in the output dropdown in the MQTT In node?

[cymplecy]

I'm keeping things simple (but explicit) - just works when using default MQTT in node settings

[zenofmud]

You can also use this as the function

var nmsg = {}
nmsg.payload = msg
return nmsg;

and this as the out function

var nmsg = {}
nmsg = msg.payload
return nmsg;

[{"id":"ee19fba1.af0b3","type":"inject","z":"ad474a87.836418","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":140,"y":100,"wires":[["fc10739f.051218"]]},{"id":"fc10739f.051218","type":"change","z":"ad474a87.836418","name":"","rules":[{"t":"set","p":"topic","pt":"msg","to":"test_stuff","tot":"str"},{"t":"set","p":"other","pt":"msg","to":"Look I'm other","tot":"str"},{"t":"set","p":"payload.animal","pt":"msg","to":"cat","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":160,"y":160,"wires":[["427154b.48e682c","baccbe3a.9ab05"]]},{"id":"427154b.48e682c","type":"function","z":"ad474a87.836418","name":"","func":"var nmsg = {}\nnmsg.payload = msg\nreturn nmsg;","outputs":1,"noerr":0,"x":150,"y":220,"wires":[["115bd288.d5810d","ee96b9f6.7a40e8"]]},{"id":"baccbe3a.9ab05","type":"debug","z":"ad474a87.836418","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":370,"y":160,"wires":[]},{"id":"ee96b9f6.7a40e8","type":"debug","z":"ad474a87.836418","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":370,"y":220,"wires":[]},{"id":"115bd288.d5810d","type":"mqtt out","z":"ad474a87.836418","name":"","topic":"test_stuff","qos":"","retain":"","broker":"70122dd0.b2a414","x":160,"y":280,"wires":[]},{"id":"d34fea20.43778","type":"mqtt in","z":"ad474a87.836418","name":"","topic":"test_stuff","qos":"2","datatype":"json","broker":"70122dd0.b2a414","x":590,"y":100,"wires":[["3b62f881.b9f0f8","733b200b.0b9ad"]]},{"id":"3b62f881.b9f0f8","type":"debug","z":"ad474a87.836418","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":790,"y":100,"wires":[]},{"id":"733b200b.0b9ad","type":"function","z":"ad474a87.836418","name":"","func":"var nmsg = {}\nnmsg = msg.payload\nreturn nmsg;","outputs":1,"noerr":0,"x":590,"y":160,"wires":[["788dcbcc.506e0c"]]},{"id":"788dcbcc.506e0c","type":"debug","z":"ad474a87.836418","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":790,"y":160,"wires":[]},{"id":"70122dd0.b2a414","type":"mqtt-broker","z":"","name":"mqttpizw","broker":"192.168.48.92","port":"1883","clientid":"","usetls":false,"compatmode":false,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthRetain":"false","birthPayload":"","closeTopic":"","closeQos":"0","closeRetain":"false","closePayload":"","willTopic":"","willQos":"0","willRetain":"false","willPayload":""}]

[Trying_to_learn]

Hi folks.

New day here. Just got up.

Wow. Ok thanks to all. @cymplecy and @zenofmud for the ideas.

Just now I need to continue with the mess from last night fixing up old mistakes from years gone by.

I shall seriously take a look at the two ideas.

Paul. in your example you use `nmsg`....... Ah! Got it.