Context Stores - maybe async but how can we tell?

[TotallyInformation]

Is there any way to programmatically detect if a context store requires asynchronous access?

Having to to handle multiple context variables is a complete pain if you have to take async access into account. The easiest way probably being to wrap the access requests in promises so that you can use Promise.all to continue your processing once all of the variables have been read.

For the most part, everyone seems to mostly assume that async access isn't required and just ignore it. Obviously though, this is not ideal.

So ideally, it would be best if we could have an API call that returned whether or not the store was async - does that exist? If not, could it be made to happen?

2nd best would be a promisified version of at least get functions to allow multiple gets to be waited on before continuing processing.

Or maybe there is an alternative that I've missed?

---

As a secondary request, would it be possible to adjust RED.util.evaluateNodeProperty for context variables to make the callback optional rather than required? If we know we don't need to handle async stores, it would make coding a lot simpler.

[drmibell]

I have no hands-on experience, but looking at the documentation for ContextStore.get(scope, key, [callback]) the description says

If no callback is provided, and the store supports synchronous access, the get function should return the individual value, or array of values for the keys. If the store does not support synchronous access it should throw an error.

That suggests that a try... catch around a get with no callback should give you an answer.

[TotallyInformation]

Doh! I missed that, thanks. Obvious short term fix anyway. Though it doesn't really help with the overall issue of suddenly having to deal with a bunch of callbacks inside your message handler function. For now, I think I'm just going to error out if this is the case and come back to it when I can get my head around the convoluted processing.

[Steve-Mcl]

I believe the better way is to always call async versions since the more exotic stores are typically async only (due to storing in a remote database or redis for example)

[drmibell]

@TotallyInformation

having to deal with a bunch of callbacks

If you need to get multiple context variables, key can be an array of keys. Presumably, the callback won't execute until they have all been assigned.

[Steve-Mcl]

@TotallyInformation

having to deal with a bunch of callbacks

You can always promisify them using node:util promisify

const setFlowContext = util.promisify(flow.set).bind(flow)
const data = await setFlowContext("my-key", "my-value", "redis")
// or 
setFlowContext("my-key", "my-value", "redis")
.then((err, data) => {
  // do stuff
}).catch(...)

[drmibell]

@Steve-Mcl

always call async versions

Agreed, although I haven't had the pleasure of working with anything so "exotic." It might be nice for someone to dummy up a simple async store for testing purposes.

[TotallyInformation]

@Steve-Mcl

I believe the better way is to always call async versions since the more exotic stores are typically async only (due to storing in a remote database or redis for example

That makes the code a lot more complex though annoyingly since I now have to move the msg processing into the callback.

@drmibell

If you need to get multiple context variables, key can be an array of keys. Presumably, the callback won't execute until they have all been assigned.

OK, that's a point. I'll have to play with that to find out how it actually returns the data and confirm that it doesn't complete until all have returned. Thanks, I think that's the best option so far.

@Steve-Mcl

You can always promisify them using node:util promisify

Yes, I realise that but while that lets me do Promise.all, it is still a lot more complex and still twists the processing into an async chain which is annoying.

@drmibell

Agreed, although I haven't had the pleasure of working with anything so "exotic." It might be nice for someone to dummy up a simple async store for testing purposes.

That's a good idea - but maybe not so simple. I've had a couple of goes at creating one but kept getting lost in the various processes. Not enough time to work it all out.

---

Thanks all, I think that having a single call with multiple keys - assuming it works, is going to be easiest.

Did I say how much I dislike JavaScript async?!

[drmibell]

@TotallyInformation

Did I say how much I dislike JavaScript async?!

Yes, but please feel free to say it as often as you like. I'm sure there are folks out there who like everything about JS, but It's a very poor fit to my little brain.

[TotallyInformation]

@drmibell

I'm sure there are folks out there who like everything about JS, but It's a very poor fit to my little brain.

Well, even the original author explained that it should never really have seen the light of day. Having been written in a hurry to prove a point. For all its foibles, Python would have been a better choice for the language of the web.

[drmibell]

@TotallyInformation

Python would have been a better choice

Another case of the first mover advantage...

[TotallyInformation]

@drmibell

Another case of the first mover advantage...

Actually of course, Python well pre-dates JavaScript. Sadly, the committee/people who were controlling it made some really basic errors of management and it languished in the dusty realms of academia for years, missing out of the rapid changes that the web was bringing. Then there was the debacle of how the version changes were managed for quite a while with a heavy-handed split between two versions that then kept developing in parallel for ages.

All of that and its embedding in Netscape Navigator and the eventual appearance of Node.js gave JavaScript an audience that became impossible to ignore.

Now JavaScript is going through its own growing pains as it seems every year brings significant additional complexity and then taking years for practical implementations to catch up.

Maybe someone needs to ask ChatGPT about creating a new language for the web.

[dceejay]

@TotallyInformation

about creating a new language

I thought the point will be that you don't need one - you just describe what you want in your native language and you get a running app... (maybe... )

[drmibell]

@dceejay

describe what you want in your native language

Imagine a world where everyone spoke JavaScript...

I remember when every new markup language was going to let you write a specification and have it instantly translated into executable code.

[TotallyInformation]

@drmibell

I remember when every new markup language was going to let you write a specification and have it instantly translated into executable code.

Actually, that's kind of how I started programming. Because COBOL was such a pain to write, Phillips had a minicomputer system where we wrote "pseudo-code" and it translated into COBOL which was then compiled - go figure!

Then later, I moved on to so called "4th generation languages", a system called "Focus" allowed us to write "natural" language queries against databases. Had the interesting trait where the query would sometimes not work so you walked back through the query to the beginning and it still wouldn't work but then you but all the same text back and it would work. Fun debugging!

And that was all back in the 1980's!

[TotallyInformation]

@TotallyInformation

@drmibell

If you need to get multiple context variables, key can be an array of keys. Presumably, the callback won't execute until they have all been assigned.

OK, that's a point. I'll have to play with that to find out how it actually returns the data and confirm that it doesn't complete until all have returned. Thanks, I think that's the best option so far.

Annoyingly, this isn't feasible in the case where you have multiple typedInput fields because the RED.util.evaluateNodeProperty does NOT have the ability to process multiple inputs. So the code continues to get more complex.

In the end, I tried Steve's promisify approach and had to make my msg input handler async all the way down.

Oddly, I tried to promisify RED.util.evaluateNodeProperty in the same way suggested for flow/global.get but I couldn't get it to work no matter what I tried - this is the simple version but I tried some more and less complex versions too:

const evalProp =
    util.promisify(RED.util.evaluateNodeProperty).bind(RED.util)
const cssSelector = await evalProp('cssSelector', node.cssSelectorType, node, msg)

Which has meant that I've had to reproduce part of RED.util.evaluateNodeProperty in my own code so that I can manually detect when an input is is a flow or global var and I can then process with:

const getFlowContext = util.promisify(node.context()[propType].get).bind(node.context()[propType])
result = await getFlowContext(contextKey.key, contextKey.store)

Everything else uses the standard evaluateNodeProperty without a callback.

To say that this is unpleasant is a significant understatement.

---

It would be fantastic if a bigger and better brain than mine could come up with a working solution to promisify evaluateNodeProperty directly.

[Steve-Mcl]

The RED.util functions dont require binding to an object as they are pure (have no this)

e.g.

            const evalProp = util.promisify(RED.util.evaluateNodeProperty)
            let data;
            try {
                data = await evalProp(node.data, node.dataType, node, msg)
            } catch (error) {
                node.error("Unable to evaluate data", msg);
                node.status({ fill: "red", shape: "ring", text: "Unable to evaluate data" });
                return;//halt flow!
            }

[TotallyInformation]

I'm sure I tried that and it didn't work ....

.... and I did, but I messed it up! Tried again with the correct input and it works! Doh

So this does work.

const evalProp = util.promisify(mod.RED.util.evaluateNodeProperty)

const cssSelector = await evalProp(node.cssSelector, node.cssSelectorType, node, msg)
const slotSourceProp = await evalProp(node.slotSourceProp, node.slotSourcePropType, node, msg)

So that has simplified things a bit.

---

Thanks everyone for your help. Not easy to get to.