How to do Black Box UI Testing of Nodes?

Developing Nodes
1

In node-red-contrib-victron we have non-trivial javascript logic in the node's html, e.g. here.

To avoid bugs and regressions, we want to add automated tests for such logic.

How to approach this?

  1. I would like to avoid browser-based testing, if possible. Is there documentation, or are there examples, on how to do this?
  2. I found node-red-node-test-helper, but I don't see how I could use this to test javascript included in the html of a node. Can I?
  3. If it has to be browser-based testing, I found this test in node-red itself; when trying to run it with grunt test-ui, it prompts me to install "UI test dependencies". The install script fails on my MacOS M1 laptop with "Only Mac 64 bits supported", which allegedly is due to outdated (deprecated, unmaintained) dependencies ("chromedriver" and others). Are these tests running and succeeding, if I am on a different architecture (e.g. linux)?
2

I can add another testing suite - unit testing which does flow-based testing, i.e. you create a flow that tests expected behaviour. Basically input message expected to produce output message. They work best as part of the Erlang-Red project.

I used those nodes to create a suite of tests for the basic Node-RED functionality. That's their intended use case which might or might not match your use case.

3

Not sure I really have much of an answer here. However, might be worth noting that for my own nodes, I split out the node's Editor JavaScript to a separate resource file and simply use a script link to load it. This has, in my view, several advantages. Firstly, I can use ES Modules instead of legacy JavaScript which gives better code isolation and access to modern syntax.

But more pertinent to this question, it would allow you to load the module to a test harness script. This wouldn't let you test everything (e.g. the node-red specific parts), it would let you do unit testing on functions - hopefully, you are breaking everything out into separate functions? You would still need to do some mocking since there is quite a bit of assumed data structures. But it should be possible.

I also break out code that is common to multiple nodes out into a separate resource file and that would also be quite testable. Check out GitHub - TotallyInformation/Node-RED-Testbed: A blank canvas to test out ideas for Node-RED custom nodes for how I do this and to see if any of it is useful to you.

Bottom line is that some things should be testable as long as:

  • You are using testable functions, not lumping processes into long code runs.
  • You can mock out some key node-red specific data and functions.
4

It's really a pity to see (pun intended), not only here but also in the backup thread, that folks seem to think in scripts and not flows.

To test the functionality of a visual tool, we use textual scripts instead of making the effort of creating a visual solution.

My approach has been to explore visual approaches first and if that fails, to fall (fail?) back to an exec node i.e. script solution or worse case makefile. The advantage is that you learn something about a new paradigm, instead of using the old and trusted textual approach. Plus a visual approach means I don't constantly switch between Node-RED and my Emacs editor - it's all done in NR.

At the end of the day, I don't really care what others do. If the trusted scripting solution plus a bit of AI is your thing, alright sounds lovely. I prefer to find a visual approach and challenge myself to think out-of-the-box, the scripting solution is always available.

Btw I also create my nodes using a node development package. So the obvious interjection: but its a node package with html + js code doesn't really apply since all my code is a flow anyway. I haven't tried adding unit tests to the flows that represent my node packages, but it wouldn't be rocket science.

5

I think you may have misunderstood this one. :slight_smile:

Here is an example of what I mean, this is the html file for the new markweb node (not yet released):

<script  type="module" async src="./resources/node-red-contrib-uibuilder/uib-markweb.js"></script>

<script type="text/html" data-template-name="uib-markweb">
    <div id="ti-edit-panel">
        .......
    </div>
</script>

As you can see, while many people put the Editor JS code for a node in the same file, I dislike that, and it makes testing hard. So in my nodes, the JS is in a resource file. Simples.

Here, we are talking about using standard web-development test tooling like Jest or similar. By all means use flows if you want. But then you may have a bunch of test flows cluttering up your node-red flows and that is not always appropriate, especially in customer situations. And even more especially if you already have existing web testing tools in use by a development team. You might be wanting your customers to use Node-RED flows but have an experienced dev team - the two do not have to be intimately connected.

6

No I think you might have misunderstood, all you are saying is if you have the right team … html code is visual … test flows cluttering …

I’m saying find a flow based solution first.

I know you’re going to point to some JS code or something and say that’s visual. It’s not what I mean and something you don’t want to understand - it seems. Fine. You like coding html and JS - I don’t, I want a visual abstraction layer.

No wonder that Node-Red hasn’t moved out of the home assistant or IoT spaces because that is all the NR can do - apparently.

  • Visual unit testing? Nope.
  • Visual code management? Nope.
  • Visual web development? Nope.
  • Visual robotics? Nope.
  • Visual code comparison? Nope.

Besides proper process management doesn’t exist in NR either, it’s just one process. So no need for process management.

But that’s fine because it’s always been that way, so no need to change anything. Steady state until n8n addresses these features.

Remember, flow based programming isn’t solely applicable to IoT, so there is nothing - going by the underlying paradigm - blocking NR from doing the things I mention above. Except a lack of imagination.

7

It is certainly possible to do end to end (E2E) tests. We use cypress in @flowfuse/node-red-dashboard (source is available in GitHub)

8

Hey thanks for your input, @gregorius and @TotallyInformation !

@TotallyInformation, we are, step by step, extracting javascript logic from the html files, to make them unit-testable.

My remaining concern is Black Box testing, though: Black box testing is, by definition, a testing approach "without peering into its [the application's] internal structures or workings". So, in practice, I want to assert correctness of test scenarios through the browser, and in an environment that (in our case) includes more than just Node-RED (it runs on a Venus device with certain libraries present, it communicates with external hardware in a test environment, etc.)

Coming back to question 3, and the test in the node-red code base: The test adds two nodes, connects them, and deploys the flow, then asserts that some debug output occurred.

How can I test such a scenario? I want to assert, through the browser, that those steps (adding nodes, connecting them, deploying) don't fail, and that I get resulting debug output.

I still want to avoid browser-based testing as much as possible, but for some scenarios it is unavoidable. So, I'm looking for help on how to do browser based testing here.

9

The UI tests in the core Node-RED code base are not maintained; they were a contribution to the project and the contributors who worked on them aren't involved any more. In a perfect world, we would of course dust them off and getting them working. But we just don't have the bandwidth in the core contributor team.

10

Thanks for the clarification, @knolleary !

@Steve-Mcl the approach you have in @flowfuse/node-red-dashboard looks very close to what I'm looking for. It's not a true black box test (because you somehow intercept API calls for deploying flows, as I understand it), but it looks close. Thanks for the pointer!

11

If you prefer full open source, TestCafe seems very popular and somewhat simpler to use than some other UI test suites. And, of course, Playwright is extremely powerful if you need to do cross-browser testing.