Developing a custom ui node for Flexdash

Developing Nodes
1

Hi @tve,

I would like to experiment with audio in the Node-RED dashboard, for another discussion to assist @nhiennguyenhuy. Would like the dashboard to be able to support an infinite stream of audio segments via the pcm-player library.

Due to a lack of time, I cannot implement this both in the old AngularJs dashboard, and the new VueJs Flexdash dashboard. So I would like to try to build my first own Flexdash ui node, that uses the pcm-player library as an npm dependency.

Would be nice if you could give me some tips about how to get started.

Thanks!!
Bart

2 Likes
2

Of course! :building_construction:

Of course... you had to pick something tricky that isn't really well supported by FlexDash, didn't you??? :roll_eyes: :roll_eyes:

Try this repo: GitHub - tve/node-red-fd-pcm-player: Node-RED node and widget to play PCM audio in FlexDash

If you're using docker, you can git clone the repo and map it into your container by adding to your command line:

  -v /home/bart/node-red-fd-pcm-player:/usr/src/node-red/node-red-fd-pcm-player \

If you're not using docker, run something like npm install /home/bart/node-red-fd-pcm-player in your node-red dir.

You will have to use the dev server (I did not build the widget for prod), so start a flexdash dev server node and go to http://localhost:1880/flexdash-dev or similar (I changed the extension from -src to -dev for the dev server).

Here are some nodes to import, they don't include the dashboard config node, so it assumes you have that (import the Hello World example from node-red-fd-corewidgets first if you don't have a dashboard node, that way you can also ensure flexdash is working first):

[{"id":"9c482194a0f045d9","type":"fd-pcm-player","z":"777070ae3efc56ba","fd_container":"f10726be88dc0ea5","fd_cols":2,"fd_rows":1,"fd_array":false,"fd_array_max":10,"name":"","title":"Pcm Player","popup_info":"","data":"","x":750,"y":140,"wires":[]},{"id":"1c0a6d92da477476","type":"file in","z":"777070ae3efc56ba","name":"read sample","filename":"/usr/src/node-red/node-red-fd-pcm-player/resources/16bit-8000.raw","filenameType":"str","format":"","chunk":false,"sendError":false,"encoding":"none","allProps":false,"x":350,"y":140,"wires":[["ceda8a1156c28b15"]]},{"id":"adcf16b317d63b30","type":"inject","z":"777070ae3efc56ba","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":150,"y":140,"wires":[["1c0a6d92da477476"]]},{"id":"ceda8a1156c28b15","type":"function","z":"777070ae3efc56ba","name":"buf2array","func":"if (Buffer.isBuffer(msg.payload)) msg.payload = Uint8Array.from(msg.payload)\nreturn {\n    source: \"16bit-8000\",\n    data: msg.payload,\n}","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":550,"y":140,"wires":[["9c482194a0f045d9"]]},{"id":"f10726be88dc0ea5","type":"flexdash container","name":"pcm-player","kind":"StdGrid","fd_children":",9c482194a0f045d9","title":"","tab":"7f846d7b8252c652","min_cols":"1","max_cols":"20","parent":"bdf2b4b185a1a993","solid":false,"cols":"1","rows":"1"},{"id":"7f846d7b8252c652","type":"flexdash tab","name":"pcm-player","icon":"mdi-music","title":"","fd_children":",f10726be88dc0ea5","fd":"e8f5aea52ab49500"},{"id":"bdf2b4b185a1a993","type":"flexdash container","name":"Container1","kind":"StdGrid","fd_children":"","title":"Cont1","tab":"99faa9f04adc062c","min_cols":"1","max_cols":"20","parent":"","solid":false,"cols":"1","rows":"1"},{"id":"99faa9f04adc062c","type":"flexdash tab","name":"Imported tab","icon":"mdi-view-dashboard","title":"Navnet","fd_children":",bdf2b4b185a1a993","fd":"e8f5aea52ab49500"}]

You will most likely have to adapt the path in the read-file node that reads the audio sample (copied from the pcm-player repo), the sample is in the resources subdir of node-red-fd-pcm-player and the path I have is for docker.

While the above works (for me) there are a bunch of issues:

  • This sends the file as one big binary buffer, there is no streaming... As you know, FlexDash is optimized for transferring state, not for streaming messages. But that's a common issue, so we need to figure out how we want to deal with that.
  • I have not looked into pcm-player to understand when it would like to be fed the next buffer. Ideally it can make a call-back when it's about to run out of data or when it has consumed one buffer. We could then feed the signal back to NR to produce the next buffer. Of course if you want to stream real-time audio then that's a different matter.
  • Right now the audio buffer is placed into the global data so all connected clients start playing. I don't know whether that's what you want or whether you want some 'play/stop' controls in the widget so individual clients can play when they want it, which is something FlexDash doesn't yet support out of the box.
  • Clients that don't have the tab with the player widget active may not play anything at all because the widget component is not loaded.

Anything else ??? :poop: :poop:

NB: writing the widget was pretty trivial, all I did was to npm install pcm-player in the widgets directory, then import PCMPlayer from 'pcm-player' and this.player = new PCMPlayer({...}). :rocket: :rocket: :rocket:

1 Like
3

Is there/should there be a naming protocol for flexdash nodes (similar to ui nodes).
I see that you mention node-red-fd-corewidgets above.

1 Like
4

I'm indeed trying to promote enforcing a node-red-fd- prefix and hence I used node-red-fd-pcm-player and if I published to NPM it would be @tve/node-red-fd-pcm-player (to remind people about the scoping...).
Thanks for the prompt!

(Edited to correct myself. Absent minded... :roll_eyes: The node-red-fd- prefix is required for the dynamic loading of widgets into the browser to take place. So it's not optional.)

2 Likes
5

You may be wondering "what now"???

If you're running the dev server, you can edit the widget .vue file (widgets/pcm-player.vue) and the moment you save it the dashboard will hot-reload the new version. Sometimes this doesn't quite work so you have to manually reload the full dashboard in your browser.

If you make changes to the widget that need to propagate to Node-RED, which is basically only if the props change, you need to run ./generate.sh at the top level of the repo to regenerate the Node-RED .js and .html files. Then you need to restart Node-RED and you need to reload the flow editor. The generate.sh script uses gen-widget-nodes.js from the @flexdash/node-red-flexdash repo, so you may have to clone that and place it next to your repo (or adjust the path in the last line of the script).

I would advise to refrain from writing a custom node for the widget at this stage, work with the auto-generated one and write a separate node (or use a function node) to produce the data, handle any output, etc. It's not as pretty and convenient right now, we can fix that later. I'd like to come up with a way to have custom code pulled in by the generator but I'm not yet sure which approach to use for that.

NB: for others that may be following and wonder how much code all this took, the source file for the widget is at node-red-fd-pcm-player/widgets/pcm-player.vue at main · tve/node-red-fd-pcm-player · GitHub and that's the only thing that I wrote other than updating package.json.

6

Really :innocent:
If I was aware of that, I would have asked you something else :lying_face:

Is it available already...
Are you are a real human, or a fork of the Github Copilot?

Ah you add the pcm-player npm package as a dependency in the package.json file of the widget, instead of the package.json file of the node. Will need to get used to that...

Could you please explain about about the magic to make the library available on the frontend side? I assume that is why you need to add the dependencies in that separate widget package.json file, so you can automatically load those dependencies are publish them to the client? So the days are over that we had to find the path of the installed library on the server, and make it available to the client via a custom endpoint. Feeling a bit spoiled now...

Yes please give me a couple of weeks to digest this information, and ask stupid questions :slight_smile:

1 Like
7

The long form explanation can be found in the docs: Developing FlexDash/Node-RED Widgets - FlexDash (that section is missing all the user-friendly how-to stuff).

The short form is that with the dev server FlexDash is served up from sources and your widgets directory is essentially 'npm installed' into the FD sources. So the browser loads your widgets and its dependencies like the rest of FD.

To use your package & widget in production you need to npm run build in the widgets dir. That will create one or two build files in the dist subdir and those get dynamically loaded through more magic.

Lots of magic... :tada:

8

FYI, that pcm-player is not suitable for streaming. It's also not a lot of code. You'll want to use the AudioContext stuff yourself... LMK if you need help.

9

Failed to do it with AudioContext at the time being. On my Android phone the distortions where really annoying. If you ever have good ideas about that, you can meet us in the other thread (in my first post above)...

Since I still didn't find time to finish my sidebar for Flexdash, I need to use start the Vite dev server via your node. But I get the error "no vite in":

image

I think you have already explained me in the past how to solve this, but unfortunately I cannot find it in my private messages...

1 Like
10

The dev node only installs the flexdash sources if they don't exist (to avoid overwriting sources you may be editing). I suspect you have an old flexdash-src dir laying around? Best is to delete it and restart the dev server (pressing that button on the node). Assuming it's configured to install FD sources you should see the status line indicating that it's doing that and you should be ready to go...

1 Like
11

I took a look at the other thread and got curious about the browser's audio api... I thought I'd try to replace that pcm-player with code that directly calls the audio API. The result is...

  • a widget that can take "streaming" audio
  • streaming means you can feed it a buffer of PCM samples and it enqueues it
  • it plays the buffers back-to-back seamlessly
  • each buffer specifies the channels, format, and sample rate, so you can mix sources, if desired...
  • at the end of each buffer it outputs a message with the number of buffers still enqueued as well as the number of milliseconds of data it has left

The resulting widget has fewer lines of code than the pcm-player thing on its own :boom:

I also tweaked the sample flow. It now reads the sample audio file, splits it up into 1 second chunks, then outputs a pile of messages with one chunk per message to simulate streaming where you might get a buffer at a time from some socket.
Then the first buffer is fed directly into the pcm-player, the second chunk is fed 500ms later, and the subsequent chunks come at 1s spacing. A debug node prints the output produced by the pcm-player widget but that info is not used in this simple demo.
(Also, I'm cutting the audio off after 8 seconds to make it easier to experiment, no need for 74 seconds...)

The widget currently prints debug info to the web browser console. It looks something like this:

PcmPlayer: converting 8000 samples from 
Int16Array(16000) [ -1014, 5831, -1169, 10705, 5769, 13117, -7071, 363, -22978, -10826, … ]
PcmPlayer: queued 1.000s of audio, start @0s
PcmPlayer: converting 8000 samples from 
Int16Array(16000) [ -2669, -12442, -6595, -11176, -3900, -13132, -4149, -19305, -3381, -9180, … ]
PcmPlayer: queued 1.000s of audio, start @1s
PcmPlayer: buffer finished, time left: 0.990
PcmPlayer: converting 8000 samples from 
Int16Array(16000) [ 4863, 619, 1977, -927, 350, -920, -2811, -4016, -7320, -8231, … ]
PcmPlayer: queued 1.000s of audio, start @2s
PcmPlayer: buffer finished, time left: 0.989
...
PcmPlayer: converting 8000 samples from 
Int16Array(16000) [ -2846, -1768, -4667, -2141, -5440, -2113, -6251, 15, -4954, 1757, … ]
PcmPlayer: queued 1.000s of audio, start @7s
PcmPlayer: buffer finished, time left: 0.990
PcmPlayer: all buffers finished

Sample flow:

[{"id":"9c482194a0f045d9","type":"fd-pcm-player","z":"777070ae3efc56ba","fd_container":"f10726be88dc0ea5","fd_cols":2,"fd_rows":1,"fd_array":false,"fd_array_max":10,"name":"","title":"Pcm Player","popup_info":"","x":910,"y":140,"wires":[["ea61f6d52a51ce68"]]},{"id":"1c0a6d92da477476","type":"file in","z":"777070ae3efc56ba","name":"read sample","filename":"/usr/src/node-red/node-red-fd-pcm-player/resources/16bit-8000.raw","filenameType":"str","format":"","chunk":false,"sendError":false,"encoding":"none","allProps":false,"x":350,"y":140,"wires":[["ceda8a1156c28b15"]]},{"id":"adcf16b317d63b30","type":"inject","z":"777070ae3efc56ba","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":150,"y":140,"wires":[["1c0a6d92da477476"]]},{"id":"ceda8a1156c28b15","type":"function","z":"777070ae3efc56ba","name":"buf2arrays","func":"// except msg.payload to be a buffer of PCM audio samples at 8kHz\n// split the buffer up into 1 second chunks\n// convert each chunk into a typed array (UInt8Array)\n// send the first chunk on the first output as an audio descriptor\n// send the remaining chunks as audio descriptors in individual messages\nif (!Buffer.isBuffer(msg.payload)) {\n    node.warn(\"msg.payload must be a buffer\")\n    return\n}\n\nconst chunk_length = 2 * 2 * 8000 // 2 channels by 2 bytes/sample by 8000Hz rate\nnode.warn(`cutting ${Math.ceil(msg.payload.length/chunk_length)} chunks`)\nconst chunks = Array(Math.ceil(msg.payload.length/chunk_length)).fill(0)\n    .map((_,ix) => Uint8Array.from(msg.payload.subarray(ix*chunk_length, (ix+1)*chunk_length)) )\nconst msgs = chunks.map(c => ({\n    source: \"16bit-8000\",\n    stream: [{\n        channels: 2,\n        rate: 8000,\n        format: 'Int16',\n        data: c,\n    }],\n}))\nnode.warn(`Got ${msgs.length} messages`)\nmsgs.splice(8, 10000) // TWEAK-ME: this throws away all but the first 8 seconds\nconst msg0 = msgs.shift()\nreturn [msg0, msgs]","outputs":2,"noerr":0,"initialize":"","finalize":"","libs":[],"x":550,"y":140,"wires":[["28eaca34691bad86"],["c5d12331b4e247a2"]]},{"id":"c5d12331b4e247a2","type":"delay","z":"777070ae3efc56ba","name":"","pauseType":"delay","timeout":"500","timeoutUnits":"milliseconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"allowrate":false,"outputs":1,"x":590,"y":180,"wires":[["6ddd9ca705faea92"]]},{"id":"6ddd9ca705faea92","type":"delay","z":"777070ae3efc56ba","name":"","pauseType":"rate","timeout":"5","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"allowrate":false,"outputs":1,"x":640,"y":220,"wires":[["28eaca34691bad86"]]},{"id":"8605977df7cfa6e0","type":"debug","z":"777070ae3efc56ba","name":"audio","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":890,"y":180,"wires":[]},{"id":"ea61f6d52a51ce68","type":"debug","z":"777070ae3efc56ba","name":"buffer ended","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":1100,"y":140,"wires":[]},{"id":"28eaca34691bad86","type":"junction","z":"777070ae3efc56ba","x":800,"y":140,"wires":[["9c482194a0f045d9","8605977df7cfa6e0"]]},{"id":"f10726be88dc0ea5","type":"flexdash container","name":"pcm-player","kind":"StdGrid","fd_children":",9c482194a0f045d9","title":"","tab":"7f846d7b8252c652","min_cols":"1","max_cols":"20","parent":"bdf2b4b185a1a993","solid":false,"cols":"1","rows":"1"},{"id":"7f846d7b8252c652","type":"flexdash tab","name":"pcm-player","icon":"mdi-music","title":"","fd_children":",f10726be88dc0ea5","fd":"e8f5aea52ab49500"},{"id":"bdf2b4b185a1a993","type":"flexdash container","name":"Container1","kind":"StdGrid","fd_children":"","title":"Cont1","tab":"99faa9f04adc062c","min_cols":"1","max_cols":"20","parent":"","solid":false,"cols":"1","rows":"1"},{"id":"99faa9f04adc062c","type":"flexdash tab","name":"Imported tab","icon":"mdi-view-dashboard","title":"Navnet","fd_children":",bdf2b4b185a1a993","fd":"e8f5aea52ab49500"}]

I did push the built file to github, so you can npm install the package and it should work in prod mode as well...

Note that this doesn't address the issues I flagged. The streaming works if you don't push it too hard. I like the way it looks in the widget but what happens on the NR side needs some enhancement.

Hope this works for you! :tada: :tada:

3 Likes
12

That indeed did the job. Thanks!!

Faster than Github Copilot :wink:
Flabbergasted...

I pulled your latest version, and restarted Node-RED. But when I import your flow, the dashboard is empty:

image

In my debug bar I see this:

"Widget node is not part of any dashboard (widget node [-> panel] -> grid -> tab -> dashboard chain broken)"

And indeed when I go to the config node, the panel property has a red border:

image

Which is a bit odd, because when I open it I don't see anything wrong:

image

Not sure whether this is because I have been experimenting some time ago with your previous flexdash versions....

13

When I create a new tab:

image

Then it seems to work:

image

But when I select your 'pcm-player' tab afterwards again, then the red border occurs again.

14

About the example flow that has been shared above: the audio quality of that streaming test is superb both on my Windows 10 portable and my Android phone.
Thorsten rocks :champagne: :+1: :partying_face:

1 Like
15

Thank you so much , @BartButenaers and @tve for your kindness!
I will try this node soon.

1 Like
16

I believe what happens is the following:

  • the imported nodes include a tab, but not a dashboard, so the tab doesn't pass validation: it's not linked to a dashboard
  • the error message could be more clear and tell you exactly what is missing
  • when you open the tab config node as you show above, the drop-down is auto-filled to the first option, which is the one and only dashboard node you have
  • you look at it and think "this is fine, why is is complaining?" and hit cancel
  • if you hit "update" instead, it would have saved what you saw and everything would have been fine

I tried to write some code that automatically links tabs to a dashboard if there is only one dashboard but I somehow stumbled but now I don't remember over what...

This is because it's the tab that is mis-configured. You need to "update" the tab.

17

That is indeed correct. If have done that now, and indeed the error is solved :+1:

18

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.