Dashboard2 - seeking help formatting a msg for the ui-audio node

Dashboard
,
1

Hello,

I am having success using Piper to generate TTS for voice notifications. I'm able to use a POST message to send text to an instance of Piper running in a Docker container, which returns the spoken text as a binary buffer.

I've tested it with node-red-contrib-play-audio, and the results are about as glorious as I could ever have hoped.

However, the next most obvious step is to wrap that buffer properly in a nice object and publish it to MQTT, and thus any device with a speaker that is subscribed to that topic will play it. I will be using this for spoken notifications in specific locations.

The node-red-contrib-play-audio node is great, but I'd also like to be able to play voice notifications on any device viewing the Dashboard, therefore ui-audio.

Where I need help is to format a ui_update.src message so a the ui-audio node is happy to accept that message as input. It wants the location of the audio as a URL ... but I have it right here in this buffer. Here. Have it. Please.

I need to somehow con ui-audio into believing that a local buffer is an URL, or ... uhhhh ... help.

Please, assuming a msg.payload of a raw buffer, please help me write a function or set up a change node such that ui-audio will play it. There has to be a field or a parameter I'm not seeing.

Simply feeding ui-audio the buffer in ui_update.src has not worked. This is a formatting thing. Halp. I stuck.

This is what I have so far:


[{"id":"6a137c41d3addec5","type":"ui-template","z":"3567e03e18126502","group":"d3197b4c4dd7b057","page":"","ui":"","name":"","order":2,"width":0,"height":0,"head":"","format":"<template>\n    <!-- if you remove the `controls` attribute, you MUST click soomething on the page before initiating play -->\n    <audio ref=\"audio\" controls autoplay src=\"\"> </audio>\n</template>\n\n<script>\n    export default {\n        watch: {\n            msg: function () {\n                if (this.msg?.topic === 'play') {\n                    if (this.$refs.audio.src !== this.msg.payload ) {\n                        this.$refs.audio.src = this.msg.payload\n                    }\n                    this.$refs.audio.play()\n                } else if (this.msg?.topic === 'stop') {\n                    this.$refs.audio.src = ''\n                } else if (this.msg?.topic === 'pause') {\n                    this.$refs.audio.pause()\n                }\n            }\n        }\n    }\n</script>\n\n<style>\n.hidden {\n    display: none !important;\n}\n</style>","storeOutMessages":true,"passthru":true,"resendOnRefresh":true,"templateScope":"local","className":"hidden","x":320,"y":340,"wires":[[]]},{"id":"caf334102ab7952d","type":"inject","z":"3567e03e18126502","name":"cantina","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"play","payload":"https://media.geeksforgeeks.org/wp-content/uploads/20240218213800/CantinaBand60.wav","payloadType":"str","x":170,"y":380,"wires":[["6a137c41d3addec5"]]},{"id":"8e135a0e9bfdccb7","type":"inject","z":"3567e03e18126502","name":"march","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"play","payload":"https://www2.cs.uic.edu/~i101/SoundFiles/ImperialMarch60.wav","payloadType":"str","x":170,"y":340,"wires":[["6a137c41d3addec5"]]},{"id":"3c290f1ea6cf8b36","type":"inject","z":"3567e03e18126502","name":"pause","props":[{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"pause","x":170,"y":300,"wires":[["6a137c41d3addec5"]]},{"id":"436efb1217534529","type":"inject","z":"3567e03e18126502","name":"stop","props":[{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"stop","x":170,"y":260,"wires":[["6a137c41d3addec5"]]},{"id":"d3197b4c4dd7b057","type":"ui-group","name":"P2 G1","page":"6685af11067a04cd","width":11,"height":"1","order":1,"showTitle":true,"className":"","visible":"true","disabled":"false","groupType":"default"},{"id":"6685af11067a04cd","type":"ui-page","name":"grid page","ui":"22ea43815413e748","path":"/page2","icon":"home","layout":"grid","theme":"70e58855f40712e7","breakpoints":[{"name":"Default","px":"0","cols":"3"},{"name":"Tablet","px":"576","cols":"6"},{"name":"Small Desktop","px":"768","cols":"9"},{"name":"Desktop","px":"1024","cols":"12"}],"order":2,"className":"","visible":"true","disabled":"false"},{"id":"22ea43815413e748","type":"ui-base","name":"base","path":"/dashboard","appIcon":"","includeClientData":true,"acceptsClientConfig":["ui-notification","ui-control"],"showPathInSidebar":false,"showPageTitle":true,"navigationStyle":"icon","titleBarStyle":"default"},{"id":"70e58855f40712e7","type":"ui-theme","name":"pink Theme","colors":{"surface":"#ffffff","primary":"#d355a5","bgPage":"#e1bcbc","groupBg":"#fbe9e9","groupOutline":"#dc8f8f"},"sizes":{"density":"compact","pagePadding":"12px","groupGap":"12px","groupBorderRadius":"4px","widgetGap":"12px"}}]
2

You can serve the audio file like any other file via an endpoint. That endpoint can be served from node-red using either httpStatic settings in your settings.js or create and endpoint using http-in-> read file -> http-response

Update. I forgot to say, you can also send the buffer instead of loading a file. You'll likely have to store it in context or generate/grab it upon request, but the concept is the same.

3

Hi @Steve-Mcl

I'm so frustratingly close to understanding what you said. Like, I get in theory, but my brain needs to see an implementation.

Let me try and restate.

In Dashboard2, audio must be played with the ui-audio node. It won't accept a buffer as input, for reasons that confuse me - surely this is arbitrary? If it can pull the buffer from an URL, why can't it just accept a buffer that's handed to it?

ui-audio must have input as an URL to where the audio is. Even though I am standing there waving a buffer at it, it wants that audio hosted somewhere instead, is the totality of what my tiny brain has grasped so far.

So I have msg.payload containing a buffer of good audio that I can already play with other methods. I just can't play it in the dash because of ... whyyyyyyyyyyy?

All I am trying to do is turn that buffer into something ui-audio will play. I've spent a day trying to do this and failing. I'm an old farmer, trying to code, and am frustrated beyond belief.

4

An endpoint is a URL - you make it serve your buffer.

Here are 2 similar solutions (casting to a google home speaker is the same deal)

1 Like
5

Hi @Steve-Mcl,

Thank you, I learned a lot, and have got it working. Your help was invaluable.

For anyone else who finds this thread via search, here is my understanding of what I learned; please correct as necessary.

If you have audio in a buffer (such as the output of a TTS ai like Piper), you cannot pass it directly to the ui-audio node, because that node needs an URL as input.

We therefore create one, using a thing called an HTTP Endpoint. The name was confusing to me, but essentially you're creating a mini-webserver that any other flow can punt a request at.

So you set up an http-in node, and you give it a URL that you like. For example, it might be /voice. That means, this node will respond on http://localhost:1880/voice, if you send a request there. You plug it directly into an HTTP-out node (which is the node that will respond when called), and forget about it.

Back to the ui-audio node. Pop it open, and in the URL field, put http://localhost:1880/voice. That's where it'll send a request when it receives a message.

Great, now back to the msg.payload that has a buffer in it.

You send that to an http-request node, configured to call the http-in endpoint we created earlier (http://localhost:1880/voice). Because the buffer is in msg.payload, it gets sent to the endpoint too, thus making that buffer the thing that will return when the URL is queried.

You connect the the output of the http-request node to the ui-audio node, which you configured to query http://localhost:1880/voice, which is now the URL of that buffer.

Tadaaaa! audio should play.

This is the flow that worked for me:

[{"id":"18236fdc807e7dd6","type":"function","z":"1b1d637ac0577302","name":"Format the message the way LocalAI likes","func":"var inputtext = msg.payload;\nmsg.payload = {\n    \"model\": \"en_GB-alba-medium.onnx\",\n    \"backend\": \"piper\",\n    \"input\": inputtext\n}\nreturn msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":400,"y":1120,"wires":[["55b6b2b54568c96f"]]},{"id":"55b6b2b54568c96f","type":"http request","z":"1b1d637ac0577302","name":"Send to Piper on LocalAI instance","method":"POST","ret":"bin","paytoqs":"ignore","url":"http://[your instance]:8080/tts","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[{"keyType":"other","keyValue":"Content-Type","valueType":"other","valueValue":"application/json"}],"x":400,"y":1180,"wires":[["ce5946b404c49809"]]},{"id":"e6cc61786434fe8c","type":"inject","z":"1b1d637ac0577302","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"This is what voice notifications sound like when generated using Piper's alba model.","payloadType":"str","x":170,"y":1120,"wires":[["18236fdc807e7dd6"]]},{"id":"325349d714f921ac","type":"http in","z":"1b1d637ac0577302","name":"","url":"/voice","method":"post","upload":true,"swaggerDoc":"","x":170,"y":1300,"wires":[["454d2974a46b1b2a"]]},{"id":"454d2974a46b1b2a","type":"http response","z":"1b1d637ac0577302","name":"","statusCode":"","headers":{},"x":350,"y":1300,"wires":[]},{"id":"ce5946b404c49809","type":"http request","z":"1b1d637ac0577302","name":"","method":"POST","ret":"bin","paytoqs":"ignore","url":"http://localhost:1880/lucy","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[],"x":730,"y":1120,"wires":[["369a44ec9d422aaa"]]},{"id":"369a44ec9d422aaa","type":"ui-audio","z":"1b1d637ac0577302","group":"0c76141b6b8548ea","name":"","order":2,"width":0,"height":0,"src":"http://localhost:1880/voice","autoplay":"on","loop":"off","muted":"off","x":940,"y":1120,"wires":[[]]},{"id":"d20b7f9e1a875f8b","type":"comment","z":"1b1d637ac0577302","name":"Injects a string. How you get your tts audio is up to you.","info":"","x":300,"y":1060,"wires":[]},{"id":"1e35b3c403beaf1f","type":"comment","z":"1b1d637ac0577302","name":"This calls the endpoint, and sends it the buffer","info":"","x":830,"y":1060,"wires":[]},{"id":"5430ea06f24919f3","type":"comment","z":"1b1d637ac0577302","name":"This is the endpoint thingy. It's what gives msg.payload a URL","info":"","x":320,"y":1260,"wires":[]},{"id":"0c76141b6b8548ea","type":"ui-group","name":"Zone arming","page":"a30cb6a10bfc3330","width":"4","height":"1","order":8,"showTitle":true,"className":"","visible":"true","disabled":"false","groupType":"default"},{"id":"a30cb6a10bfc3330","type":"ui-page","name":"Newdash","ui":"cf9dc2ba60d1396c","path":"/page6","icon":"home","layout":"grid","theme":"7ba40ffa60b22682","breakpoints":[{"name":"Default","px":"0","cols":"4"},{"name":"Tablet","px":"576","cols":"6"},{"name":"Small Desktop","px":"768","cols":"9"},{"name":"Desktop","px":"1024","cols":"12"}],"order":1,"className":"","visible":"true","disabled":"false"},{"id":"cf9dc2ba60d1396c","type":"ui-base","name":"My Dashboard","path":"/dashboard","includeClientData":true,"acceptsClientConfig":["ui-notification","ui-control"],"showPathInSidebar":true,"navigationStyle":"default","titleBarStyle":"default"},{"id":"7ba40ffa60b22682","type":"ui-theme","name":"Default Theme","colors":{"surface":"#ffffff","primary":"#0080ff","bgPage":"#eeeeee","groupBg":"#ffffff","groupOutline":"#cccccc"},"sizes":{"pagePadding":"2px","groupGap":"12px","groupBorderRadius":"4px","widgetGap":"4px","density":"comfortable"}}]
1 Like
6

Nooooo I have broken something!

Please have a look at my flow; there must be something obvious I am not seeing.

[{"id":"b9765ce47c15b7ae","type":"debug","z":"1b1d637ac0577302","name":"First hurdle","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":710,"y":300,"wires":[]},{"id":"df41fe923050cae5","type":"debug","z":"1b1d637ac0577302","name":"Second hurdle","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":340,"y":520,"wires":[]},{"id":"203876051c371051","type":"debug","z":"1b1d637ac0577302","name":"Third hurdle","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":1170,"y":300,"wires":[]},{"id":"99915e6780a51e6e","type":"inject","z":"1b1d637ac0577302","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"This is what voice notifications sound like when generated using Piper's alba model.","payloadType":"str","x":250,"y":180,"wires":[["61a1f51e26df5f13"]]},{"id":"e7ff0621bb935849","type":"http in","z":"1b1d637ac0577302","name":"","url":"/voice","method":"post","upload":true,"swaggerDoc":"","x":310,"y":400,"wires":[["df41fe923050cae5","5c90a491d9d9b7ff"]]},{"id":"5c90a491d9d9b7ff","type":"http response","z":"1b1d637ac0577302","name":"","statusCode":"","headers":{},"x":530,"y":400,"wires":[]},{"id":"301523ffa21de59d","type":"ui-audio","z":"1b1d637ac0577302","group":"0c76141b6b8548ea","name":"","order":0,"width":0,"height":0,"src":"http://localhost:1880/voice","autoplay":"on","loop":"off","muted":"off","x":1240,"y":260,"wires":[[]]},{"id":"61a1f51e26df5f13","type":"function","z":"1b1d637ac0577302","name":"Format the message","func":"var inputtext = msg.payload;\nmsg.payload = {\n    \"model\": \"en_GB-alba-medium.onnx\",\n    \"backend\": \"piper\",\n    \"input\": inputtext\n}\nreturn msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":300,"y":260,"wires":[["46b45af3b681a9d6"]]},{"id":"46b45af3b681a9d6","type":"http request","z":"1b1d637ac0577302","name":"Send to Piper","method":"POST","ret":"bin","paytoqs":"ignore","url":"http://192.168.2.75:8080/tts","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[{"keyType":"other","keyValue":"Content-Type","valueType":"other","valueValue":"application/json"}],"x":520,"y":260,"wires":[["b9765ce47c15b7ae","a5ed50a96ffe5256"]]},{"id":"a5ed50a96ffe5256","type":"http request","z":"1b1d637ac0577302","name":"","method":"POST","ret":"bin","paytoqs":"ignore","url":"http://localhost:1880/voice","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[],"x":950,"y":260,"wires":[["301523ffa21de59d","203876051c371051"]]},{"id":"e90eb6ec2ac6b923","type":"comment","z":"1b1d637ac0577302","name":"1. Everything is fine until here","info":"","x":700,"y":220,"wires":[]},{"id":"02c6732b4a8078ef","type":"comment","z":"1b1d637ac0577302","name":"2. when this is called, the payload arrives empty","info":"","x":560,"y":460,"wires":[]},{"id":"0c76141b6b8548ea","type":"ui-group","name":"Zone arming","page":"a30cb6a10bfc3330","width":"4","height":"1","order":8,"showTitle":true,"className":"","visible":"true","disabled":"false","groupType":"default"},{"id":"a30cb6a10bfc3330","type":"ui-page","name":"Newdash","ui":"cf9dc2ba60d1396c","path":"/page6","icon":"home","layout":"grid","theme":"7ba40ffa60b22682","breakpoints":[{"name":"Default","px":"0","cols":"4"},{"name":"Tablet","px":"576","cols":"6"},{"name":"Small Desktop","px":"768","cols":"9"},{"name":"Desktop","px":"1024","cols":"12"}],"order":1,"className":"","visible":"true","disabled":"false"},{"id":"cf9dc2ba60d1396c","type":"ui-base","name":"My Dashboard","path":"/dashboard","includeClientData":true,"acceptsClientConfig":["ui-notification","ui-control"],"showPathInSidebar":true,"navigationStyle":"default","titleBarStyle":"default"},{"id":"7ba40ffa60b22682","type":"ui-theme","name":"Default Theme","colors":{"surface":"#ffffff","primary":"#0080ff","bgPage":"#eeeeee","groupBg":"#ffffff","groupOutline":"#cccccc"},"sizes":{"pagePadding":"2px","groupGap":"12px","groupBorderRadius":"4px","widgetGap":"4px","density":"comfortable"}}]

Right now, when I press Inject, I do get a valid buffer back. The debug at First Hurdle shows exactly what I expect: a msg.payload that's a nice buffer. If I point that to a Play Audio node, I can hear the voice; that part works.

It breaks when it goes through the HTTP endpoint, and I need to understand why: this is where my understanding gets weak.

The debug at Second Hurdle is in the middle of the HTTP endpoint. This debug shows that the payload arrives empty. I don't understand how.

What went wrong between those two points?

debug
debug301Ă—554 20.5 KB

This is the debug output. I am so confused right now.