# How to pass audio chunks through a Node-RED flow

**URL:** <https://discourse.nodered.org/t/how-to-pass-audio-chunks-through-a-node-red-flow/3723>\
**Category:** Core Development\
**Created:** [5 October 2018 21:24 UTC](https://discourse.nodered.org/t/how-to-pass-audio-chunks-through-a-node-red-flow/3723 "2018-10-05T21:24:06Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [5 October 2018 21:24 UTC](https://discourse.nodered.org/t/how-to-pass-audio-chunks-through-a-node-red-flow/3723/1 "2018-10-05T21:24:06Z")

</div>

Hi folks,

I started experimenting with audio for this [discussion](https://discourse.nodered.org/t/play-sound-from-usb-microphone/2551), and now I need to create some extra audio-nodes (e.g. an audio analyser, a volume amplifier ...). However there are _multiple ways to pass audio samples between those nodes_, and I have no clue which is the best one. The gates have been opened for **voting** 🤨

**A) What is audio**

Raw audio (PCM) is nothing more than chunks of **_bytes_** , representing series of audio samples (see [here](https://github.com/bartbutenaers/node-red-contrib-wav-headers#more-detailed) for some basic explanation). However some EXTRA information need to be added, to describe the content of those bytes:

- _ **Number of channels** _: the bytes can come from a single microphone, but also e.g. from two microphones (left sample / right sample / left sample / ...).
- _ **Number of bits** _: each sample can consist of a number of bits (8, 16, 24, 32 ...).
- _ **Sample rate** _: since the sample contain only an (amplitude) value and no timestamp, we need to indicate how many samples are generated per second.

The raw audio bytes can only be interpreted correctly by a receiver (node), if that receiver is aware of this information.

**B) How to pass audio in a flow**  
It seems to me that audio can be passed between nodes (as messages) in the following ways:

1. **Only pass the raw audio bytes in msg.payload**. For example the [node-red-contrib-micropi](https://www.npmjs.com/package/node-red-contrib-micropi) nodes generate output messages that contain ONLY raw audio (PCM) samples:

2. **Pass raw audio bytes in msg.payload and add 3 extra fields to the message**.

3. **Pass raw audio bytes and 3 extra fields together in msg.payload (as WAV)**. WAV is well known audio container, which is in fact a series of headers (44 bits length in total) followed by the raw audio bytes.

4. **Pass raw audio bytes and 3 extra fields together in msg.payload (as AudioBuffer instance)**. I want to create extra Node-RED nodes for audio manipulation, and the best option (in plain Javascript) seems the [audio.js](https://github.com/audiojs) project (which contains a large series of NodeJs nodes). Those nodes are all based on the **AudioBuffer** class. However when we would pass the raw bytes as normal Buffers in Node-RED (see three previous options), then EACH of the Node-RED audio nodes should do this:

I think I would vote for option 2, and use option 3 only when e.g. the audio chunk needs to be stored as WAV. But I would love to have the AudioBuffer from option 4, but I don't now how to integrate that in Node-RED ...

Thanks !!!  
Bart

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [5 October 2018 21:51 UTC](https://discourse.nodered.org/t/how-to-pass-audio-chunks-through-a-node-red-flow/3723/2 "2018-10-05T21:51:46Z")

</div>

Some good thinking there - so I will now jump in and demonstrate my ignorance 🙂

Would (say) a function node be able to do anything to an AudioBuffer ? How easy would it be to half the volume for example ? Or how would/could other (non-audio) nodes be able to "join in" and help this flow ?

If other nodes wouldn't be able to do things to a payload of type audiobuffer - then maybe put some data useful to the flow in payload and the audiobuffer in another msg.property (that all your audio nodes would use), and then things like switch node can act on the audio to route it ? or... err (I did warn you I was making this up) . Also would audiobuffer then need a special file writer node - or would you just have a "convert back to mp3/wav/..." node then to file ?

WAV (3) is of course appealing - but how well does that stack up playback wise - ie do they "join" back into a stream ok ?  
There is no reason of course why you couldn't do 3 with the extra data of 2 alongside so other nodes could act on the metadata without looking into the stream

(and of course happy to consider changes to the audio out node(s) to help accommodate whichever route is works out best)

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [5 October 2018 22:50 UTC](https://discourse.nodered.org/t/how-to-pass-audio-chunks-through-a-node-red-flow/3723/3 "2018-10-05T22:50:00Z")

</div>

> [@dceejay](#):
>
> how would/could other (non-audio) nodes be able to "join in"

Good point. I 'think' a lot of **non-audio nodes** won't be able to use the AudioBuffer instance _decently_:

- A **function node** should do a require of all the used audio.js nodes (via `functionGlobalContext`), so that is perhaps not user-friendly...
- Don't think a **switch node** is able to use the [information](https://www.npmjs.com/package/audio-buffer) in the audioBuffer instance.
- Join node might result in strange effects?
- ...

I had only looked at the **audio-related** nodes. So perhaps it is a bit confusing to users when I pass AudioBuffer instances ..,

> [@dceejay](#):
>
> How easy would it be to half the volume for example ?

Haven't tested all my code yet, since I didn't know if AudioBuffer was a good option...  
But I assume an AudioVolume node would be as simple as this:

```auto
	var AudioAmplifier = require('audio-gain');

	function AudioVolumeNode(config) {
		RED.nodes.createNode(this, config);
        this.volume = config.volume || 1;

        var node = this;
        
        var options = { volume: node.volume };
        node.audioAmplifier = AudioAmplifier(options);

        node.on("input", function(msg) {                
            // Let's assume the input message contains an AudioBuffer instance ...
            var audioBuffer = msg.payload;

            // Send the amplified audio chunks to the output
            msg.payload = node.audioAmplifier.process(audioBuffer.channelData);
            
            return msg;
        });
    }

```

It is rather simple this way, and in plain Javascript (so no C++ builds required). That is why I wanted to use AudioBuffers, to avoid having to program all the functionality by myself.

Remark : I haven't done any performance tests yet!

> [@dceejay](#):
>
> maybe put some data useful to the flow in payload and the audiobuffer in another msg.property

Hmm, now we are talking 🤔 Although when talking about sound, this 'sounds' like a workaround (since the 3 information fields will be available twice in the message). But it might indeed do the job since the non-audio nodes can use the normal message fields, and the audio-related nodes use the field with the AudioBuffer. Good proposal !!!!!

> [@dceejay](#):
>
> would audiobuffer then need a special file writer node

Haven't tested it yet, but I would indeed assume (like e.g. in Java) that the AudioBuffer instance will need to be serialized to a text in the file? And when reading the file, the text has to be deserialized again to an AudioBuffer instance... Will need to play with it, because I haven't used that ever in Javascript. Must admit that I don't like the idea that I would have to create extra file nodes. Should work automatically somehow with the normal file nodes ... Of course they can always store a **.wav file** :

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/2X/6/6a7978b178b665b3aa496244e37405ccddf651be.png)

> [@dceejay](#):
>
> WAV (3) is of course appealing - but how well does that stack up playback wise

Could you please explain that a bit more? Google-translate exploded when I entered your sentence ...

> [@dceejay](#):
>
> and of course happy to consider changes to the audio out node(s)

That is what I was hoping 🤣  
Thanks !

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [5 October 2018 23:10 UTC](https://discourse.nodered.org/t/how-to-pass-audio-chunks-through-a-node-red-flow/3723/4 "2018-10-05T23:10:36Z")

</div>

Re WAV. I mean. If a large file is turned into chunks. And each chunk is its own bit of wav, with wav headers etc, then how well do they stick back together ? Is it seamless to the ear ? And likewise can they just be written sequentially to make one big file ? (Or do you need to strip all the intermediate headers?)

Re audiobuffer to file, I guess you would have to have some conversion anyway, (to mp3 or wav etc) and that would then be a normal buffer, which existing file node would handle.

And don't forget we already have the concept of parts metadata used by the split join and other nodes

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [6 October 2018 07:37 UTC](https://discourse.nodered.org/t/how-to-pass-audio-chunks-through-a-node-red-flow/3723/5 "2018-10-06T07:37:38Z")

</div>

> [@dceejay](#):
>
> how well do they stick back together

Wav buffer = buffer with headers (44 bytes) + buffer with raw data. When you want to join multiple wav's into a single larger wav, indeed you need to _ **remove the intermediate headers** _ of all the wav's that you append. With option 2 more standard Node-RED nodes can be used, since you just need to use my wav-headers node only at the end of the audio manipulation.

> [@dceejay](#):
>
> audiobuffer to file, I guess you would have to have some conversion anyway, (to mp3 or wav etc) and that would then be a normal buffer, which existing file node would handle.

Think you are right: send the message to some raw-audio-to-mp3 node, which converts the audiobuffer to mp3. Now the msg.payload contains a normal buffer with mp3 data that can be written to a file. Since the file-out node ignores the `msg.property` field (containing the audiobuffer instance), there is NO issue (of audiobuffer being serialized). And after the file-in node we need to add the create-audiobuffer node to recreate the audiobuffer again from the loaded raw audio.

> [@dceejay](#):
>
> And don't forget we already have the concept of parts metadata used by the split join and other nodes

Never used those nodes. Do you mean I could use the `msg.parts` somewhere for audio?

> [@dceejay](#):
>
> Is it seamless to the ear ?

Certainly not. That is why I have so much troubles to play the wav chunks _ **decently** _ in the dashboard...

> [@dceejay](#):
>
> put some data useful to the flow in payload and the audiobuffer in another msg.property

There is one drawback of this approach: the raw audio data is now passed twice through every wire, since raw audio is inside the `msg.payload` AND inside `msg.property`. So when you have 2 wires connected to an output, Node-RED will clone both buffers. Again waste of resources, both memory and cpu.

Thanks for discussing...

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [6 October 2018 08:24 UTC](https://discourse.nodered.org/t/how-to-pass-audio-chunks-through-a-node-red-flow/3723/6 "2018-10-06T08:24:21Z")

</div>

So yeah that makes using the WAV approach a pain... probably not efficient, or certainly less appealing than it was.

The split node (and I think file in and csv nodes) add a `msg.parts` property that contains things like the `index` of that chunk to help when putting back together later... if you are slicing things up it may be useful to follow that pattern.

Not sure why you say the data gets passed twice - I was thinking the audiobuffer would be in (say) `msg.audio` and ONLY the useful (to other nodes) metadata would be in `msg.payload` and not the raw data as it had now been converted to the audiobuffer.

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [6 October 2018 08:51 UTC](https://discourse.nodered.org/t/how-to-pass-audio-chunks-through-a-node-red-flow/3723/7 "2018-10-06T08:51:41Z")

</div>

> [@dceejay](#):
>
> and not the raw data as it had now been converted to the audiobuffer.

Ah ok, thought you ment both raw audio in `msg.payload` AND audiobuffer in `msg.property`. When only one of those, there indeed it will be **no** cloning overhead...

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [6 October 2018 10:07 UTC](https://discourse.nodered.org/t/how-to-pass-audio-chunks-through-a-node-red-flow/3723/8 "2018-10-06T10:07:44Z")

</div>

well - if you fork the flow then it will always clone one of them - (but yeah - better than duplicating everything as well)

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [6 October 2018 12:31 UTC](https://discourse.nodered.org/t/how-to-pass-audio-chunks-through-a-node-red-flow/3723/9 "2018-10-06T12:31:57Z")

</div>

I am very far from being even a novice at buffers. However, this has me thinking that Node-RED as it stands may not be very good at handling buffers because of its focus on msgs?

So that had me thinking whether there might not be a better way to integrate them as streams? NodeJS is good at handling streams - I think?

So I wonder - pure speculation on my part, I've no idea whether this is at all feasible - whether it wouldn't be better to have some stream processing nodes that didn't use the normal msg passing but used - for example - PIPES? They _should_, I think, be much better at handling streams.

I know that would introduce a secondary communication method but I think that it should then be expandable to other stream processing which would make Node-RED a lot more useful for handling high-volume data throughput.

Anyway, a random thought that had been swirling through my noggin for a while so I thought I'd put it out there for someone to shout down. Or not.

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [6 October 2018 13:54 UTC](https://discourse.nodered.org/t/how-to-pass-audio-chunks-through-a-node-red-flow/3723/10 "2018-10-06T13:54:42Z")

</div>

Hi mister Knight,

> [@TotallyInformation](#):
>
> better way to integrate them as streams

As long as you don't fork the flow, I don't think there is an issue. As soon as you add a fork, the messages will be cloned. But that is not necessary a bad thing, if you want to do somerhing else with the cloned data. E.g. one audiostream needs to be amplified, and the other one not.

**But for high data volumes we need to have a single chain of nodes** , without forks. But that means that every node should pass all its input messages on its output. For example I need to create an AudioAnalyzer node that calculates the frequency spectrum. But it should APPEND this data in the `msg.spectrum` field of the original input message, and pass that UPDATED message on its output port. This way we can avoid a lot of forks..

Did I forgot something?

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [6 October 2018 14:35 UTC](https://discourse.nodered.org/t/how-to-pass-audio-chunks-through-a-node-red-flow/3723/11 "2018-10-06T14:35:08Z")

</div>

Bart, don't worry. Julian is talking about a whole different paradigm for the flow within Node-RED. If all inputs and outputs were node.js streams then they could be wired together and data would just flow... hopefully 🙂 In practise it make more sense when you are coding lines of code as you can just "dot" them together js style input.read.manipulate.write sort of thing - where you don't then have forks or cloning going on.

Anyway - the file in and out nodes do indeed try to handle streams - and what we found is that even on an in stream then node.js effectively does chunk it up into pieces to hand to the next thing. The chunk size depends on the operating system. Net is - that once chunked we may as well just pass it on as a chunk anyway (which is what the file in node does when set to send chunks) - so meh... not going to worry about it all too much at this point.

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [7 October 2018 11:28 UTC](https://discourse.nodered.org/t/how-to-pass-audio-chunks-through-a-node-red-flow/3723/12 "2018-10-07T11:28:06Z")

</div>

> [@dceejay](#):
>
> Anyway - the file in and out nodes do indeed try to handle streams - and what we found is that even on an in stream then node.js effectively does chunk it up into pieces to hand to the next thing. The chunk size depends on the operating system. Net is - that once chunked we may as well just pass it on as a chunk anyway (which is what the file in node does when set to send chunks) - so meh... not going to worry about it all too much at this point.

Cool. And interesting. Thanks for explaining. One of many corners of NodeJS I don't understand very well.

---

<div class="post-metadata">

**Author:** ![donpedro](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/donpedro/32/78837_2.png) [@donpedro](https://discourse.nodered.org/u/donpedro)\
**Post date:** [18 December 2023 13:40 UTC](https://discourse.nodered.org/t/how-to-pass-audio-chunks-through-a-node-red-flow/3723/13 "2023-12-18T13:40:22Z")

</div>

This is on a slightly different topic, but I ran across this thread when looking for people who had tried what I was about to try: node streams in Node-RED. I didn't really find anything more on it, so I've done my own thing; I've just published my first nodes so I guess I'm now in very early beta, but I'd love some input!

The existing project is [gulpetl](http://gulpetl.com), and we're adding Node-RED capabilities to a project already built around streams. It's for ETL rather than audio, so design choices are different, but we are running into the types of gotchas you touched upon like message cloning. So far we're sailing through!

Here's the [Node-RED section](https://docs.gulpetl.com/node-red) of our docs, with Getting Started and what-not. I hope to get a brief tour finished up in the next couple of days.