Batch node 'weirdness' (or not!)

[mudwalker]

Hi,

I am seeing some 'weirdness' when using the Batch Node and think that this is due to my misunderstanding of how it functions. I have an array X elements long, I want to batch this up into groups of six so...

Number of elements in array = 8
Number of Messages = 6; Overlap = 5;
Batch 0 = Array[0] to Array[5]
Batch 1 = Array[1] to Array[6]
Batch 2 = Array[2] to Array[7]

So far, so good...
Batch 3 = Array[3] to Array[there is no element at 8]

So, my question is - Are values generated beyond Batch 2 as we have run out of elements to include in the Batch, or does it continue until it reaches the end of the array with the last Batch consisting just of the last element?

TIA
Colin

[nisiyama]

@mudwalker

Are values generated beyond Batch 2 as we have run out of elements to include in the Batch, or does it continue until it reaches the end of the array with the last Batch consisting just of the last element?

The batch node create sequences so that the number of messages in the sequence to be equals to configured size. Therefore, in this case, Batch 2 will be the last sequence.

[mudwalker]

Thank you for that, it is what I understood from reading and what I was hoping for, but not what I am always seeing. I now know that I need to look elsewhere for the problem.

[dceejay]

Can you paste a simple example flow that shows the problem so we can check / and or explain. Thanks

[mudwalker]

Thank you for the offer @dceejay, it is much appreciated. I am always astounded at the help available on this Forum.

At the moment, I am learning a lot about arrays, and Node Red in general, trying to fault find this. As much as I would love to get this sorted, I have a 'learning opportunity' which although frustrating to some extent, hopefully means I will come out stronger!! I think I may have race conditions (several parallel paths doing very similar things) or, more likely, played in areas I don't yet quite understand! I have some things I still want to try.

Bottom line is, if you want the flow for research, I will post it. If not, I appreciate the offer, but can I take a rain check for the moment and come back a while later if I should need further help?

I make a commitment to post what I find if I fix it!!

[dceejay]

Hi,
you're most welcome to work it out yourself... one less thing for us to worry about
But yes we're here if you get stuck or it doesn't make sense. Of course there may indeed be a bug - in which case @nisiyama would like to ensure we get it fixed. Thanks

[mudwalker]

@nisiyama

In the interests of speeding the diagnosis of any possible bug, I have attached my flow, but please be aware, I am no Javascript programmer, so apologies for any mess!

This code is my modified take on bits and pieces discussed in Best strategy for Reading & Sorting Electricity Cost Array.

Explanation:
The problem happens when we get to the end of the available data. This data refreshes every evening after about 18:00 hours for the next 24 or so hours. If there is plenty of data present, all appears OK, but this may be due to the cheaper slots being earlier in the day. But when generating longer slots by combining say six array elements (relating to 30 minute slots with their pricing) to create a longer slot based on costs, and 'running out' of data, the problem starts to become apparent.

Let me use the 3 hour slot as the example (six array elements). The last slot available in our data is from 22:30 - 23:00, I can see 3 hour slots that are available from after 20:30. Looking at various times, I have seen the array length to be variable, sometimes correct, sometimes too many objects in the array. When the number of objects is correct, all is OK, when there are too many for the time left available, that is when the problem starts, hence the initial question.

If you need any further information, or I have confused you, please do not hesitate to ask. Apologies in advance for any basic mistakes I have made (currently check variable types!).

A big thank you for your efforts.

HTH

Octopus flows.json (38.8 KB)

Edited with updated flow to ease display of data.

[mudwalker]

@nisiyama
I think I may have just found the problem. It is mostly likely me! Will let you know.

[mudwalker]

@nisiyama

I have been cleaning up things and now have a flow that will return 12 or 17 elements from an array with a fixed set of data as input (below). I eliminated as much as I could from the flow to keep things manageable, and the flow runs in it's own tab.

I have also taken a screen shot showing my screen. The first debug message comes after a 'Restart Flows', the second message immediately after, it remains at 17 elements after that.

Please let me know if you need anything else.

[{"id":"39b20d7.7470672","type":"tab","label":"Batch Node Test","disabled":false,"info":""},{"id":"f24cf3bb.225c18","type":"batch","z":"39b20d7.7470672","name":"Batch 6 periods","mode":"count","count":"6","overlap":"5","interval":10,"allowEmptySequence":false,"topics":[],"x":180,"y":140,"wires":[["c66cf133.eee248"]]},{"id":"c66cf133.eee248","type":"join","z":"39b20d7.7470672","name":"","mode":"auto","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":330,"y":140,"wires":[["389d8b77.ae9cf4"]]},{"id":"7631bb2e.b9001c","type":"function","z":"39b20d7.7470672","name":"Troublesome array","func":"msg.payload = [{\"value_exc_vat\":5,\"value_inc_vat\":5.25,\"valid_from\":1588541400000,\"valid_to\":1588543200000},{\"value_exc_vat\":5.88,\"value_inc_vat\":6.174,\"valid_from\":1588539600000,\"valid_to\":1588541400000},{\"value_exc_vat\":5.88,\"value_inc_vat\":6.174,\"valid_from\":1588537800000,\"valid_to\":1588539600000},{\"value_exc_vat\":7.46,\"value_inc_vat\":7.833,\"valid_from\":1588536000000,\"valid_to\":1588537800000},{\"value_exc_vat\":6.8,\"value_inc_vat\":7.14,\"valid_from\":1588534200000,\"valid_to\":1588536000000},{\"value_exc_vat\":7.14,\"value_inc_vat\":7.497,\"valid_from\":1588532400000,\"valid_to\":1588534200000},{\"value_exc_vat\":7.64,\"value_inc_vat\":8.022,\"valid_from\":1588530600000,\"valid_to\":1588532400000},{\"value_exc_vat\":8.06,\"value_inc_vat\":8.463,\"valid_from\":1588528800000,\"valid_to\":1588530600000},{\"value_exc_vat\":20.4,\"value_inc_vat\":21.42,\"valid_from\":1588527000000,\"valid_to\":1588528800000},{\"value_exc_vat\":20.4,\"value_inc_vat\":21.42,\"valid_from\":1588525200000,\"valid_to\":1588527000000},{\"value_exc_vat\":20.4,\"value_inc_vat\":21.42,\"valid_from\":1588523400000,\"valid_to\":1588525200000},{\"value_exc_vat\":19.24,\"value_inc_vat\":20.202,\"valid_from\":1588521600000,\"valid_to\":1588523400000},{\"value_exc_vat\":19.31,\"value_inc_vat\":20.2755,\"valid_from\":1588519800000,\"valid_to\":1588521600000},{\"value_exc_vat\":17.88,\"value_inc_vat\":18.774,\"valid_from\":1588518000000,\"valid_to\":1588519800000},{\"value_exc_vat\":5.54,\"value_inc_vat\":5.817,\"valid_from\":1588516200000,\"valid_to\":1588518000000},{\"value_exc_vat\":5.21,\"value_inc_vat\":5.4705,\"valid_from\":1588514400000,\"valid_to\":1588516200000},{\"value_exc_vat\":5.04,\"value_inc_vat\":5.292,\"valid_from\":1588512600000,\"valid_to\":1588514400000}];\nreturn msg;","outputs":1,"noerr":0,"x":330,"y":100,"wires":[["21a98821.b254d8"]]},{"id":"118eccdd.e0f5b3","type":"inject","z":"39b20d7.7470672","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":140,"y":100,"wires":[["7631bb2e.b9001c"]]},{"id":"21a98821.b254d8","type":"split","z":"39b20d7.7470672","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":490,"y":100,"wires":[["f24cf3bb.225c18"]]},{"id":"6c201645.253fc","type":"join","z":"39b20d7.7470672","name":"Create Array","mode":"custom","build":"array","property":"test","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"2","count":"","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":330,"y":180,"wires":[["d88af334.d083e"]]},{"id":"d88af334.d083e","type":"debug","z":"39b20d7.7470672","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":490,"y":180,"wires":[]},{"id":"389d8b77.ae9cf4","type":"change","z":"39b20d7.7470672","name":"Transform","rules":[{"t":"set","p":"test.start","pt":"msg","to":"payload[5].valid_from","tot":"msg"},{"t":"set","p":"test.time_end","pt":"msg","to":"payload[0].valid_to","tot":"msg"},{"t":"set","p":"test.value","pt":"msg","to":"payload[0].valid_to","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":480,"y":140,"wires":[["6c201645.253fc"]]},{"id":"c605b2b4.73fa4","type":"comment","z":"39b20d7.7470672","name":"Batch Node Test","info":"","x":140,"y":54,"wires":[]}]

[nisiyama]

@mudwalker The following are the situations that are occurring.
The length of the array in question is N=17.
In the first run, the internal queue for batch (length M[6]) is empty, so N-M+1 (=12) sequences are generated. Next time you give the same input, the internal queue of batch is filled with the last M messages of the first run. Therefore, N (17) sequences are generated after 2nd run.
You can reproduce similar situation with following flow:

[{"id":"6ee5ee49.5891b","type":"inject","z":"ecc5dabf.b26c38","name":"","topic":"","payload":"[0, 1,2,3,4]","payloadType":"json","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":140,"y":400,"wires":[["f6eb1fa3.2005c"]]},{"id":"f6eb1fa3.2005c","type":"split","z":"ecc5dabf.b26c38","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":310,"y":400,"wires":[["b3a88e11.5b7d3"]]},{"id":"43fd3dc8.834b94","type":"join","z":"ecc5dabf.b26c38","name":"","mode":"auto","build":"string","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":"false","timeout":"","count":"","reduceRight":false,"x":590,"y":400,"wires":[["492ad561.3ae2ac"]]},{"id":"b3a88e11.5b7d3","type":"batch","z":"ecc5dabf.b26c38","name":"","mode":"count","count":"3","overlap":"2","interval":10,"allowEmptySequence":false,"topics":[],"x":450,"y":400,"wires":[["43fd3dc8.834b94"]]},{"id":"492ad561.3ae2ac","type":"debug","z":"ecc5dabf.b26c38","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":760,"y":400,"wires":[]}]

[mudwalker]

@nisiyama

Thank you for your quick response and explanation. I understand what is happening, your flow illustrates it precisely.

So my next question is, how do I reset the buffer so that we start from '0' again?

[nisiyama]

@mudwalker Current batch node assumes a single continuous input sequence and does not have a reset feature.
As a workaround, you can filter messages composed from mixed sequences as shown in the following flow.

[{"id":"6ee5ee49.5891b","type":"inject","z":"ecc5dabf.b26c38","name":"","topic":"","payload":"[0, 1,2,3,4]","payloadType":"json","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":160,"y":400,"wires":[["f6eb1fa3.2005c"]]},{"id":"f6eb1fa3.2005c","type":"split","z":"ecc5dabf.b26c38","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":310,"y":400,"wires":[["b3a88e11.5b7d3"]]},{"id":"43fd3dc8.834b94","type":"join","z":"ecc5dabf.b26c38","name":"","mode":"auto","build":"string","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":"false","timeout":"","count":"","reduceRight":false,"x":590,"y":400,"wires":[["b772446a.2d0d38","1744a74c.946cb9"]]},{"id":"b3a88e11.5b7d3","type":"batch","z":"ecc5dabf.b26c38","name":"","mode":"count","count":"3","overlap":"2","interval":10,"allowEmptySequence":false,"topics":[],"x":450,"y":400,"wires":[["43fd3dc8.834b94"]]},{"id":"492ad561.3ae2ac","type":"debug","z":"ecc5dabf.b26c38","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":990,"y":440,"wires":[]},{"id":"b772446a.2d0d38","type":"function","z":"ecc5dabf.b26c38","name":"Filter Out Mixed Array","func":"function isMixed(a) {\n    if (a.length > 1) {\n        var prev = a[0];\n        for(var i = 1; i < a.length; i++) {\n            var v = a[i];\n            if (prev > v) {\n                return true;\n            }\n        }\n    }\n    return false;\n}\n\nlet payload = msg.payload;\nif (!isMixed(payload)) {\n    return msg;\n}","outputs":1,"noerr":0,"x":780,"y":440,"wires":[["492ad561.3ae2ac"]]},{"id":"1744a74c.946cb9","type":"debug","z":"ecc5dabf.b26c38","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":750,"y":400,"wires":[]}]

[nisiyama]

@dceejay For a long term solution, I'd like to add the ability to initialize the internal state of batch node with the reset property of a message similar to join node. What do you think?

[mudwalker]

Thank you @nisiyama, your response is very much appreciated.

I am now playing with integrating your suggested solution to my application.

[dceejay]

Yes makes sense to add that ability. Need to check what the behaviour should be (or makes sense) if both a payload and reset arrive on the same msg

[nisiyama]

In the implementation of the join node, if the reset is specified, the payload is not processed. To be consistent with this, it should be processed in a similar way. I think that reset will be processed differently than the normal data processing path, this behavior seems to be reasonable.