I need to publish a list (object really) of topics to an MQTT broker -- is there a way to publish multiple topics in one shot, or is the best bet to run my list through a split node coupled with a publish node?
Best to use split. Or, if that is proving hard, just do a small function.
split works, just thinking of shooting 100 topics to the broker possibly several times a second -- don't want to deal with timeout issues or have to rely on delays and things to get everything to play nice -- that's why I was hoping to shoot everything over in one message.
If your issue is that you don't want one topic to be out of sync with another topic, then you could bundle all the messages into one JSON string - send that on a single topic.
Disadvantage is that receiving clients would have to decode the string to find what they want
YMMV
You can deal with timeouts fairly readily but if you need tight timings and high volume, that could turn out to be an issue. You will want to test things and you might consider splitting workloads across host devices to make sure that you don't hit any processor, memory and disk I/O bottlenecks.
It is hard to predict if you will have issues as so much depends on what else is happening, how busy things are and how much data you are shifting.
With MQTT as with any other interface, you should always be thinking about how data will be processed so, as Simon says, you might consider aggregating some of the data if it makes sense.
I'm afraid that you will need to do some testing in order to work out the best approach for your situation.
1 Like
Posting results of my case for anyone who comes searching. Running NR on a 1 GHz Cortex-A8 (ARM).
Before adding MQTT publishes, NR was about 14-24% CPU.
After adding MQTT publishes, split up by topics, NR was 28-60% CPU 
Decided to send everything as one JSON object together in one-shot: 20-24% CPU with occasional spikes to 30-31%.
Mosquitto (MQTT broker) never went above 1% CPU.
One-shot JSON is better in many ways and requires much less to prepare, and I don't like the CPU %'s for split-by-topic, so going with one-shot. As Jiro would say, "Ultimate simplicity leads to purity."
1 Like
I would be interested to knowing how many items were you publishing each time and how often?
Is NR also subscribing to those topics?
Here's the one-shot, didn't bother to count how many entries are inside:
{"Test.AvgPress":[0,"psig"],"Test.AvgTemp":[0,"F"],"Test.DO":[0.875,"g/cm3"],"Test.DW":[0.999,"g/cm3"],"Test.Step":[40,""],"Test.StepTime":[5441.219,"s"],"Test.StepVolume":[0,"bbl"],"Test.Well":["Default",""],"Test.GrossLiq":[0,"bbl"],"Test.ActOilNOC":[0,"bbl"],"Test.ActWaterNOC":[0,"bbl"],"Test.StdOilNOC":[0,"bbl"],"Test.StdWaterNOC":[0,"bbl"],"Test.ActOil":[0,"bbl"],"Test.ActWater":[0,"bbl"],"Test.StdOil":[0,"bbl"],"Test.StdWater":[0,"bbl"],"Test.StdGas":[-0.03768494234268098,"MSCF"],"Test.ActCutNOC":[0,"%"],"Test.StdCutNOC":[0,"%"],"Test.ActCut":[0,"%"],"Test.StdCut":[0,"%"],"Test.GrossLiqRate":[0,"BPD"],"Test.ActOilNOCRate":[0,"BPD"],"Test.ActWaterNOCRate":[0,"BPD"],"Test.StdOilNOCRate":[0,"BPD"],"Test.StdWaterNOCRate":[0,"BPD"],"Test.ActOilRate":[0,"BPD"],"Test.ActWaterRate":[0,"BPD"],"Test.StdOilRate":[0,"BPD"],"Test.StdWaterRate":[0,"BPD"],"Alarm.HighP":[0,""],"Alarm.HighT":[0,""],"Alarm.HighDriveLevel":[0,""],"Alarm.HighSoS":[0,""],"Alarm.HighGain":[0,""],"Alarm.LowSNR":[0,""],"UIT-AF01.VolTotal.GrossLiq":[0,"bbl"],"UIT-AF01.VolTotal.ActOilNOC":[0,"bbl"],"UIT-AF01.VolTotal.ActWaterNOC":[0,"bbl"],"UIT-AF01.VolTotal.StdOilNOC":[0,"bbl"],"UIT-AF01.VolTotal.StdWaterNOC":[0,"bbl"],"FIT-AF02.VolTotal.StdGas":[-0.03768494234268098,"MSCF"],"LIT-AF03.PV":[0,"%"],"PIT-AF04.PV":[0,"psig"],"TT-AF05.PV":[0,"F"],"LCV-AF06.PV":[100,"% closed"],"LCV-AF07.PV":[100,"% closed"],"UIT-AF01.VolFlow":[0,"BPD"],"UIT-AF01.MassFlow":[0,"lb/h"],"UIT-AF01.Temp":[77.31441772460943,"F"],"UIT-AF01.Density":[0.000050000000745058064,"g/cm3"],"UIT-AF01.DriveLevel":[5.735607147216797,"%"],"UIT-AF01.SensALevel":[79.96589660644531,"%"],"UIT-AF01.SensBLevel":[79.06885528564453,"%"],"UIT-AF01.TubeFreq":[264.48681640625,"Hz"],"UIT-AF01.2PhaseSig":[7.006492321624085e-42,""],"UIT-AF01.VolTotal":[0,"bbl"],"UIT-AF01.MassTotal":[0,"lb"],"FIT-AF02.VolFlow":[-0.4635500841199167,"MACFD"],"FIT-AF02.Velocity":[-0.3352603359328793,"ft/s"],"FIT-AF02.SoS":[1138.401381925648,"ft/s"],"FIT-AF02.Gain":[60,"dB"],"FIT-AF02.SNR":[42,"dB"],"FIT-AF02.VolTotal":[-0.006836107869609553,"MACF"],"UIT-AF01.InstWaterCut":[-675.1159108402285,"%"]}
So before, everything you see in there was split so that it was one item per publish. NR is not subscribing to anything (yet).
And how often was that being published?
Oh right, sorry, once per second.
That is only 66 publishes/second when split. I am surprised it consumed that much processor. Did you check by stopping just the publish that it was the publish that was causing the cpu usage?
Is it a Pi you are running on? If so which?
Yes, it's the publish(es). No, this is a FreeWave ZumIQ board with a 1 GHz Cortex-A8, single core. It would probably be nothing on a Pi 3 or 4.
OK, that makes sense then.
