Node red dashboard audio wish

#1

Dont know were to post this on github to ask for it so i try it here.

I love the ui_audio node to play audio prompts.
But for a new project i would love to get a start and end/stop when starting playing audio and when playing is finished.
Is it possible to get this in payload or as node status?

Thanks
Rogier

0 Likes

#2

...Also it would be nice to be able to stop a sound mid-play (ex: sending msg.reset).

1 Like

#3

Hey Dave (@dceejay),
Both proposals make sense, especially when live audio streaming would ever see the daylight ...
I had already implemented an experimental version of the msg.reset (or msg.stop or whatever...) since I needed to stop somehow the endless silent audio source that I start behind the scenes.

Do you have any suggestions about both proposals? Then I will take them into account, if you want. Maybe it would be useful to implement these feature requests first, since I didn't manage yet to get rid of the disturbance in the live audio stream on Android...

Thanks!
Bart

1 Like

#4

Bart, always happy to look at a PR.
Reset no problem.
What is the use case for stopped and start ? Would doing that via Status (and then status node) be OK ?

Must admit not been following your progress so closely. Can you (just about) handle streams of raw packet ?

0 Likes

#5

for me the start stop would be to control the lights during playback.
when it start lights down, when finished lights up :wink:

1 Like

#6

Guys, can we summarize it like this:

  • Input message msg.reset allows the current sound to be stopped, whether it is a single audio file or an audio stream.
  • Output message msg.stop=true is send when the current sound is stopped. I assume this might be handy e.g. when you play an mp3 file. As soon as the file is entirely played, a message is send to make sure the flow can respond e.g. by playing the next file?
  • Output message msg.start=true is send when the current sound is started. This might be a bit less usefull (I think) because the flow already knows that the sound is started, since the flow has just now send an audio chunk to the dashboard...

About the web audio stream. @GChapo has already been testing it a lot and the current status is this (correct me Glenn if not correct!):

  • On desktop browser (like e.g. Chrome) the quality is acceptable.
  • On Android the sound has lots of disturbances. I still have no clue why...
  • On iPhone there is no sound at all. But I think this is due to their autoplay policy. Will need to fix that also, but cannot debug it since I have no iPhone...

But I have some other private stuff that I need to finish, before I continue with the streaming (because it is taking much more time as expected ...).

1 Like

#7

on osx the audio is perfect!

0 Likes

#8

Hey Rogier,

The web audio stream is another thing I'm working on. Currently the audio-out node only allows you to play a single audio fragment (e.g. an mp3 file). And that will probably work nicely on osx.

However we would like it to be able to play an infinite stream of audio fragments, e.g. from a microphone. So I have to play all these fragments seamless, which is not as easy as it looks. The least disturbance between the individual fragments will be very annoying for your ears. And on mobile phones (Android and iPhone) I didn't manage yet to play seamless streams. But that is another pull-request, so sorry for going off-topic ...

0 Likes

#9

Not sure I want to see an output pin on this node. I'd prefer the status to reflect the state of playback.

1 Like

#10

Hey Dave, the status update is indeed a nice feature!
Is there any particular reason, you don't want the audio-out node to have an output? Is it because the audio-out node doesn't accept user input, so you find it confusing? But it could be useful to have an output message when the audio fragment has been finished by the client, so you can e.g. send a new audio fragment for playback.

0 Likes

#11

You could use a status node and set it to the audio note :wink:
That is what i have in mind to use it to trigger other things

1 Like

#12

What @RogierQ just said :-)...
I just refer nodes that are logically the end of a flow to look like they end it...

I'd also like to retro fit some of this thinking into some of the non-dashboard audio nodes that may use the right edge button for mute etc.

0 Likes

#13

Ok, I can clearly see your point from a visual point of view in the flow editor.

But a question about the Status-node: Currently I have only used the node status to display something visual in the editor, but I wasn't aware that these statusses are also passed through the flow (instead of messages) to exchange information between nodes. From some discussions I had concluded that we were only supposed to exchange data via the 'visible' wires, and that other 'invisible' data exchange was not allowed in the Node-RED philisophy?

Is there perhaps any rule of thumb for node developers? When something changes that the flow should be aware of: when should we send an output message and when should we update the node status. Perhaps I haven't done this correctly in my other contributions :woozy_face:

Thanks !!

0 Likes

#14

Hi Bart,

Status messages aren't passed through the flow. They are indeed mainly for visual effect, BUT... they also create an event that the status node receives (and can filter on). This is similar to the Catch node that catches node.error events.

Status is intended to reflect the status of the flow (or node) itself - rather than the state of the message - though this is blurred :-)... So it is intended for things like the state of a network connection, the depth of a queue, if a node is active or inactive, rather than the value of the payload.... of course in a function node you can set node.status to anything so this can be abused fairly quickly :-0...

And yes it is fairly arbitrary. If it's something where you would "normally" expect it to be part of a flow eg http-request - then yes it makes sense to also handle the error as part of the output message. If it's something you have handed off to something else then status may be more appropriate. But somethings that fail a lot... do mean we change our minds... like the file output node now has an output.

Also of course if the node has an output button (like debug) then it shouldn't have a output pin as well.

2 Likes

Adding extra development documentation
#15

Hey guys,

Before I submit a pull request, it would be nice to get some 'constructive' feedback.
Here is a demo, since you can not easily install my personal dashboard version:

Some explanation of the demo:

  • When pressing 'start' an mp3 fragment is send to the audio-out node. Once the web audio has started playing the fragment in the browser, the dashboard will send status 'started' back to the Node-RED flow.
  • Since I couldn't add sound to this animated gif, you can watch the equalizer bar in the right screen to see whether the audio is playing or not.
  • As soon as the mp3 fragment has been played 'entirely' by the web audio, the dashboard will send status 'completed' back to the Node-RED flow.
  • When pressing 'stop' during playback of the mp3 fragment, the web audio playback will be stopped. And the dashboard will send status 'reset' back to the Node-RED flow. P.S. the demo shows status text 'resetted', but I have changed the text afterwards to 'reset'.

In the info tab of the audio-out node, some extra text has been added:

image

I have tested this on Chrome Windows and Chrome android.

Remark: does anybody expect extra issues in case multiple clients are listening simultaneously to the same audio?

Bart

3 Likes

#16

Looks perfect to me! Thanks!
Does it also work with tts?
Thats my main use of the audio node

0 Likes

#17

@BartButenaers I'd say send a blank and no blob for end of playing - eg like the preceding http request - so it is cleared when finished. This also leaves the possibility free for "stopped" if there was ever to be a stop command, or maybe "paused" etc.... Also I'd prefer "playing" instead of started... :slight_smile:

0 Likes

#18

If you send a blank, is it possible to catch that change then with the status node?
So do i have a trigger that tells me it has stopped?

0 Likes

#19

actually can be an empty string ""...
image

1 Like

#20

Nope. Seems that I will need to handle the speech synthesis differently:

  • It should be stopped this way.
  • And to detect the end of the speech playback, I think I should use this event.

But for some reason I get no sound at all using TTS, so a bit difficult to test :woozy_face: Don't know what is wrong, because I have it on all my Raspberry's (so not a result of my dashboard changes)...

Don't know if my demo was not clear, but when you send a msg.reset=true then the playback will already be stopped. So that functionality is also part of this pull request ... Do you prefer perhaps msg.stop instead of msg.reset?

In the current version I haven't implemented pause or resume:

  • Web audio can only be started or stopped. However when 'pause' or 'resume' is required, we would need to remember ourselves the time offset (where we have interrupted the fragment). Like in this Codepen example .
  • On the other hand, the speech synthesis seem to have pause and resume methods out of the box.
0 Likes

Playing mp3 Audio