[ANNOUNCE] Visual programming a function node with Blockly - Feedback request


#41

Some changes on the Github version:

  • Two new categories:
    image

  • Type checking has been removed from node_context_get and node_context_set blocks, since ANY kind of data can be stored on memory. Now you can e.g. get a boolean variable from memory and check whether its value is 'true':
    image

  • The edit field on the node_send block has been replaced by an extra input:
    image
    This way the output number can be determined dynamically, which offers much more flexibility to the user.

About the last point: Blockly offers shadow blocks which are blocks that can be added by default, to offer some kind of default value (in this case output number '1'). The nice thing about such shadow blocks is that they are added automatically: both in the toolbox and when you drag them in the workspace. You cannot remove them, so the default value will always be availalbe. You can change the default value, or drag another block on top to replace the shadow block. And when you remove your new block, the the shadow box will automatically be available again (so you ALWAYS have a value). See demo:

blockly_shadow2

Must admit that Julian pointed me in the good direction with Blockly. So I will forgive him that he has completely demolished my social life the last month...
Found this comparison with Scratch, and Blockly seems indeed the best match for what I wanted to achieve in combination with Node-Red:
image

Now I have to go shopping with the wife and kids ...


#42

I'm just away for the weekend but really looking forward to getting stuck in with this next week :slight_smile:


#43

Haha! Please ask your wife and kids to forgive me :blush: Enjoy the shopping.

Brilliant job. A great addition to Node-RED.


#44

Not published to npm yet? Looks good enough to publish.


#45

Hi folks,

Before version 0.0.1 becomes available in the grocery near your home, I would like to add timer support to it. However there are multiple ways to implement this in my contribution, so hopefully you guys can give me some advice to help me with all my questions !!!!!

Blockly offers a "delay" block, that pauses the JS interpreter which runs their generated Javascript code. However we will run the javascript in a sandbox on top of NodeJs (similar to the function node), so we will have to build our own solution based on setInterval and clearInterval.

1 - BASIC TIMER BLOCKS
Some basic timer blocks have been added to this Node-Red contribution, to support multiple ways of creating timers. For example log a text 4 times, with 2 seconds in between:
image

And a similar one, for example ro repeat periodically until a condition is true:
image

The advantage of these basic blocks is that they are very EASY to use! However they have a series of disadvantages, which are described below...

2 - STOP OBSOLETE TIMERS
As soon as a message arrives in the node, the generated Javascript code will be executed. This means the timer will be started, and run until a specified condition is fullfilled. However there will be other circumstances to stop the timer, otherwise multiple timers would be started (which would be running simultaneously):

  • The timer should stop automatically when the flow is (re)deployed. It seems that this already working fine, I 'assume' because the sandbox is restarted ...
  • The timer should be stopped explicit when a new input message arrives. Thought to do this somehow automatically (to make sure the users don't forget to stop it), but sometimes you want to start the timer e.g. when the msg.start=true and stop it when the msg.stop=true.

To accomplish explicit stopping a 'specific' timer, I assume I need something like this:
image

But this means that the timer name should be specified, when the timer is created. So the basic block should be extended with the timer name:

image

But now the block becomes more difficult to understand... Any other suggestions?

3 - TIMER ID'S
As can be seen in the first screenshot, the names of the variables (representing the timer id's) are random generated (e.g. timerId_045lwr09cnj). Reason is that those names should be unique, in case multiple timers are created:

image

However this isn't sufficient in case e.g. the timer is created inside a loop:
image

In this example two timers are created, but the timer id variable will hold the id of the last timer. So I assume that when the first timer is stopped, that it will incorrectly stop the last timer ...

Could solve that by generating (at the top of the Javascript code) a function, since each function call has its own timerId variable:

function repeatForCount(count, intervalMillis, callbackFn) {
    var i = 0;
    var timerId = setInterval(function() {
        callbackFn();
        if(++i === count) {
            clearInterval(timerId);
        }
    }, intervalMillis);
}

repeatForCount(4, 5000, function() {
    // ...
});

Is this readable enough for people that want to learn by looking at the generated Javascript code? Or better proposals?

4 - TIMING CONFUSING
Suppose our user adds two timers, as in in the screenshot in the third paragraph. I 'assume' that this might be misleading, since he might think that the first timer will run for 8 (= 4 x 2) seconds and AFTERWARDS the second timer will run for 8 seconds. Or am I just getting paranoid ...

However the first timer will be started, and IMMEDIATELY afterwards the second timer will be started. This means both timers will run simultaneously, and both timers will be finished after 8 seconds.

Don't know how to visualise this better. I could add an extra input for statements that are executed AFTER the timer has finished, something like this:
image

Or should the user determine by himself when the timer is being stopped, like in this example:
image

5 - OTHERS
Any other suggestions, issues, ... about timers?

Thanks again !!!!
Bart


#46

Bart, first of all, I think this is really cool - LEGO for NR

As a "novice" user I think I would also expect the default behavior as you stated:

To create this default behavior, would it be possible to "add code inside" that makes the timer wait to start until an eventual previous timer has ended? But invisible to the user since this would then be default behavior?

And if the user really wants parallel execution, could there be a setting for "run IMMEDIATELY" (maybe we find a better phrase)?

A related question: let's assume you have defined several blocks of timers with other code in between. Like

timer0
timer1
block1
block2
block3
timer2
timer3
timer4

I would also expect waiting timers should only wait for their closest neighbour to finish

Well, just my feeling & input
Kind regards, Walter


#47

Wow, there is a lot going on there! Well done for thinking things through and laying out the issues for us.

My immediate thought is whether it would be better to treat timers as discrete functions. That's to say, make the use create a timer function and then call that function in the main code - would that ease the complexity? Doesn't really solve any of those problems but might make them more amenable to comprehension by beginners? A timer function could then include the optional "afterwards" which I like as a concept. Might even include a "before" though that might just be confusing, not sure of the use case there. This approach would also be easy to mirror in the code.

Would it then also be possible to use a different colour to indicate a time function? With a tooltip reminding people that calling a timer function returns immediately. I think that I would also want to see some text clearly showing that this is a timer and not a loop, the orange loop structure would be very confusing I think since timers are always going to work somewhat differently. So rather than "Repeat 4 times every 2 seconds", "Timer repeats 4 times every 2 seconds" or "Timer will repeat 4 times every 2 seconds"?

Another advantage of having the timer as a function might be that you could separate out the various options without confusing the main thrust of the logic. After all, a timer is a parallel piece of logic.

In terms of stopping timers, yes this is always a hard problem. It is hard in JS itself. It certainly needs to be a separate option I think as it may only rarely need to be surfaced.

In terms of stopping timers when a new msg arrives, I think I disagree here. I wouldn't expect that to happen. If we reinforce the message that timers are independent pieces of logic - fire and forget as it were - I think that is the more common use.

By making each timer a separate block of logic, this is reinforced and you can then simply have a calling option that returns the timer id and another block for doing timer cancels.

Just my initial thoughts anyway.


#48

Whoaa!

I think its still very much under development

Totally up to Bart but I don't think its a general release stage


#49

I'm a simple person so I'd never dream of writing a timer in JS inside a function and I'm wondering how many people would need to do this?

Basically I don't think its an initial release issue but could be done later down the line

My basic wish would be for the Blocky node to be able to do all the stuff I currently use JSONata to do
Simon


#50

Hi guys,

Thanks for thinking out loud. Sorry for the late reply, but lots of familly stuff the next weeks ...

@krambriw: the problem in NodeJs is that I cannot simply wait/sleep as in other languages like Java. If I would wait, Node-Red would have to wait for me. So Node-Red would become unresponsive, which is not what we want...

The only thing I can do is executing the next statements asynchronous, i.e. in a callback function. In your example this would result in something like this:

timer0 is started with a callback function that will start the next statement (timer1) when finished
--> timer1 is started with a callback function that will start the next statement (block1) when finished
--> block1
--> block2
--> block3
--> timer2 is started with a callback function that will start the next statement (timer3) when finished
--> timer3 ...

I have asked this morning on the Blockly forum whether something like this is possible, but I doubt ...

@TotallyInformation: hopefully I have interpreted your thougths correctly...
So you like the 'afterwards' statements, which is called via a callback function automatically. So I should generate something like this:

function createTimer(callback) {
    var timerId= setInterval(function() {
        // Execute all statements that should be runned periodically ...

        if( /* check condition that marks end of the timer */ ) {
            clearInterval(timerId);
            callback();
        }
    }, /*timeout time*/);
}

createTimer(function() {
    // The callback function ...
    // Add the next statements, that should be called AFTER the timer has ended
});

By the way, the timers are in a separate toolbox category. So I could give them e.g. a red color, and a tooltip.
image

Could you explain a bit more why you wouldn't stop a timer when a new message arrives. And do I understand correctly that you want to see somewhere in blockly the timerId's being visualised so we can manually stop a specific timer:
image

If you would have some free time left, it would be great if you could draw a quick 'blockly alike' example of what you mean (in paint or whatever)!

@cymplecy: I'm considering of skipping the timers from version 0.0.1, and add them in version 0.0.2 (since I have still a lot of questions and large part of the community is on holiday at the moment). If some people could test this contribution (without the timers), I could publish it in a couple of weeks on NPM.


#51

Before I can release my first version, there is another question that should be solved:

The generated Javascript (based on the blocks that have been dropped in the Blockly editor), will be executed as soon as an input message arrives in the blockly node. However in the Blockly editor it is currently not visible when this logic is being executed.

Moreover it also possible to execute logic (in a function node), when the node is closed (e.g. when the flow is deployed):

node.on('close', function() {
    // tidy up any async code here - shutdown connections and so on.
});

This is currently also not supported...

I could solve this by adding automatically following block to the editor (and make sure this block cannot be deleted by the user!)

image

Other applications based on Blockly do it the same way. For example in Openhab you can program your own rules. However you can also draw your rules using Blockly:
image
This way it becomes immediately visible where you have to add your logic and when it will be executed.

I think we should do something similar. And we should do it already in version 0.0.1, to avoid backwards compatibility problems ...

Again all thoughts are welcome !


#52

I think that is sensible.

It isn't that uncommon when writing flows that deal with the real world. But neither is it common either. It is certainly a more advanced function area. But if the process can be simplified using Blockly, that would be a tremendous gain for anyone learning JavaScript in general and Node-RED specifically - in my view, for what it is worth. Timers give you some measure of freedom from the single-threaded nature of NodeJS.

At a quick look, I think that I'd thought you would realise the code as something like:

function uniqueAfterTFcallbackName = function(){
...
})

function uniqueTimerFnName = setInterval(function(uniqueAfterTFcallbackName ) {
...
});

Though, forgive me because I've written that completely off the top of my head so it might be wrong. Hope it makes sense though.

But that is just my thought and I certainly wouldn't claim to be the worlds greatest JavaScript programmer!

Your clear timer block would then be separate again and would consume the timer id. To be honest, I was more thinking about simplifying the blocks rather than the JavaScript though.

I wouldn't stop a timer automatically based on an incoming msg because that is too restrictive and you are making a massive assumption about how everyone wants to use the capability. Given that timers are specifically to create some logic that runs in parallel to your main thread, automatically cancelling it when the next msg arrives may not be what is wanted. But if it is wanted, with the ability to easily manually cancel, a user can easily write the logic - they are not forced to though and can let multiple timers start and expire at will.

Apologies that I've only picked this up this morning and I have to rush to work. Busy week this week but if I get any time, I'll see if I can draw something.

Maybe I'm a simple bear but I would assume, given the nature of Node-RED, that the Blockly node would work like the function node. That's to say that it would execute as soon as the msg hits it and the logic would end when it ends. The cleanup in on close executes on redeploy or on node-red shutdown and I don't see why a consumer of the node would ever need to see what is in it? They wouldn't ever normally see it and don't in the function node.

What does need to go in there is some logic to delete any timers but unless you are creating any event listeners, TCP/UDP ports or similar, I can't think of anything else that would be needed.


#53

As I've said - I'm a simple person :slight_smile:

I use standard nodes 1st- contrib nodes second and then either a bit of JSONata or a little bit of JS in a function node

I'd like the Blockly node to let me not have to write JSONata or JS

So advanced stuff like code that executes on node closing is way beyond anything I envisage using it for

I think what you are doing is so new to Node-RED that your bound to have to break things as development goes on so I wouldn't totally worry about it at this stage - people know that when version number says 0.0.1 they are getting alpha/beta code :slight_smile:
Simon


#54

Ok - first attempt to replace one of my working function nodes with blockly one

This is my existing node
image

I'm trying to code the 1st line but I cannot get the true/false block to drop into the slot - it just gets rejected


#55

updated this morning but still says
image

Still voting for just "send" :slight_smile:


#56

If we're being very picky, its Node-RED not Node-Red


#57

I've spent the last few years teaching myself to write it properly! :slight_smile:


#58

Simon,

are you sure you have the last Github version installed?

The version on Github should allow the get ANY type of values from memory, even booleans. And when you look at the Github code for the node_send block, you will see that the 'msg' is removed. There is now only the texts "send" and "to output" (but not anymore "msg") :

Blockly.Blocks['node_send'] = {
    init: function() {
        this.appendValueInput("MESSAGE_INPUT")
            .setCheck('Object')
            .setAlign(Blockly.ALIGN_LEFT)
            .appendField('send');        
        this.appendValueInput("OUT_NR_INPUT")
            .setCheck('Number')
            .setAlign(Blockly.ALIGN_LEFT)
            .appendField('to output');       

So I think you are experimenting with an older version ... Perhaps you should try to uninstall it, and install it again.


#59

I don't seem to be able to uninstall it - it was still there when I restarted Node-RED

C:\Users\Simon\.node-red\node_modules\fs-ext>if not defined npm_config_node_gyp (node "C:\Program Files\nodejs\node_modules\npm\node_modules\npm-lifecycle\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" configure build )  else (node "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js" configure build )
Building the projects in this solution one at a time. To enable parallel build, please add the "/m" switch.
  fs-ext.cc
  win_delay_load_hook.cc
..\fs-ext.cc(108): warning C4996: 'Nan::NanErrnoException': was declared deprecated [C:\Users\Simon\.node-red\node_modules\fs-ext\build\fs-ext.vcxproj]
  C:\Users\Simon\.node-red\node_modules\nan\nan.h(887): note: see declaration of 'Nan::NanErrnoException'
..\fs-ext.cc(153): warning C4996: 'Nan::Callback::Call': was declared deprecated [C:\Users\Simon\.node-red\node_modules\fs-ext\build\fs-ext.vcxproj]
  C:\Users\Simon\.node-red\node_modules\nan\nan.h(1568): note: see declaration of 'Nan::Callback::Call'
..\fs-ext.cc(195): error C3861: 'fcntl': identifier not found [C:\Users\Simon\.node-red\node_modules\fs-ext\build\fs-ext.vcxproj]
..\fs-ext.cc(297): warning C4996: 'Nan::NanErrnoException': was declared deprecated [C:\Users\Simon\.node-red\node_modules\fs-ext\build\fs-ext.vcxproj]
  C:\Users\Simon\.node-red\node_modules\nan\nan.h(887): note: see declaration of 'Nan::NanErrnoException'
..\fs-ext.cc(339): warning C4996: 'Nan::NanErrnoException': was declared deprecated [C:\Users\Simon\.node-red\node_modules\fs-ext\build\fs-ext.vcxproj]
  C:\Users\Simon\.node-red\node_modules\nan\nan.h(887): note: see declaration of 'Nan::NanErrnoException'
..\fs-ext.cc(374): error C3861: 'fcntl': identifier not found [C:\Users\Simon\.node-red\node_modules\fs-ext\build\fs-ext.vcxproj]
..\fs-ext.cc(375): warning C4996: 'Nan::NanErrnoException': was declared deprecated [C:\Users\Simon\.node-red\node_modules\fs-ext\build\fs-ext.vcxproj]
  C:\Users\Simon\.node-red\node_modules\nan\nan.h(887): note: see declaration of 'Nan::NanErrnoException'
..\fs-ext.cc(433): warning C4996: 'Nan::NanErrnoException': was declared deprecated [C:\Users\Simon\.node-red\node_modules\fs-ext\build\fs-ext.vcxproj]
  C:\Users\Simon\.node-red\node_modules\nan\nan.h(887): note: see declaration of 'Nan::NanErrnoException'
gyp ERR! build error
gyp ERR! stack Error: `C:\Program Files (x86)\MSBuild\14.0\bin\msbuild.exe` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onExit (C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\lib\build.js:258:23)
gyp ERR! stack     at emitTwo (events.js:126:13)
gyp ERR! stack     at ChildProcess.emit (events.js:214:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:198:12)
gyp ERR! System Windows_NT 10.0.17134
gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "configure" "build"
gyp ERR! cwd C:\Users\Simon\.node-red\node_modules\fs-ext
gyp ERR! node -v v8.9.4
gyp ERR! node-gyp -v v3.6.2
gyp ERR! not ok
npm WARN node-red-project@0.0.1 No repository field.
npm WARN node-red-project@0.0.1 No license field.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fs-ext@0.5.0 (node_modules\fs-ext):
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fs-ext@0.5.0 install: `node-gyp configure build`
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: Exit status 1

up to date in 14.714s

#60

That doesn't ring a bell... Can only say that you should always refresh your browser window (when updating to a new version of it), because most of this contribution is client-side coding.