That's what I have tried to use at first, but it did not work as expected.
I'll do more tests first ...
Because, if I'm not mistaken, flow is not a reference to node.context().flow.
OOOPS -- the "old" deprecated memory only methods for use in functions are
context.flow.myvar and context.global.myvar
(and indeed context.node.myvar for context available just in that node.)
[ not just flow.myvar as mentioned previously... - been so long since I used them I forgot... ]
There seem to be some deeply flawed ideas being proposed here for a very small (if any) gain.
- Direct access only works for memory storage.
- There is no future guarantee even that will continue to work.
- You end up using a different process for memory storage to other storage and that is pretty much bound to cause you confusion eventually.
@TotallyInformation I disagree. Especially if we want to make new users life easier.
Also, I've realised yesterday:
- The normal
flow.set("myVar", myVar)is converting all myMap()type variables toObject - Which is causing error, if I try to use Map() functions later!!!
(Took me 3 days of debugging to realise that.
I had to delete all flow.set(...) occurences in every node one by one.
IMHO it is BASIC behaviour in every programming language that if I store a value in a variable, it will not be altered during the process.
At this point I've started to really hate Node.js .
Also this problem would be solvable by using my idea.
This may give you some ideas javascript - How do you JSON.stringify an ES6 Map? - Stack Overflow
There are LOTS of JavaScript objects that cannot be serialised. This is not a Node-RED issue per-se. You may have seen my recent posts on the subject. It is the reason that Node-RED filters out much of the output from the http-in and some other similar nodes (they contain circular references).
It is for some of these reasons that recent versions of uibuilder now contain several tools for more safely serialising complex JavaScript objects that JSON.serialise (and hence the set/get functions) cannot handle.
Maps, Sets, Circular references and more cannot be handled without additional work. Try this function node code for example:
const defaultData = {
standard: {
string: "R1C1",
"odd-prop-name": 42,
"i-am": true,
"date": new Date(),
"array": [1, "a", 2, true, 3],
},
functions: {
myCustomFn: function aDifferentName() { return true },
arrowFn: () => { return true },
asyncFn: async function aDifferentName2() { return true },
asyncArrowFn: async () => { return true },
generatorFn: function* mygeny() { yield 'Hello' },
asyncGgeneratorFn: async function* mygeny2() { yield 'Goodby' },
},
specials: {
"null": null,
"undefined": undefined,
"NaN": NaN,
"Infinity": Infinity,
"bigint": 9007199254740992n, // Larger than Number.MAX_SAFE_INTEGER
"symbol": Symbol("mySymbol"),
// @ts-ignore
"map": new Map([['key', 'value'], ['key-2', 42]]),
"set": new Set([1, 2, 3]),
"weakmap": new WeakMap([[{ id: 2 }, 'metadata']]),
"weakset": new WeakSet([{ id: 1 }, { id: 2 }]),
"regex": /test/gi,
"error": new Error('Process failed'),
"arrayBuffer": new ArrayBuffer(8),
"uint8array": new Uint8Array([10, 20, 30]), // typed array
"url": new URL('https://example.come/some/path?x=this,y=that'),
"promise": new Promise(() => { }),
},
// Circular reference example
circular: {}
}
defaultData.circular.self = defaultData
msg.payload = RED.util.uib.saferSerialize(defaultData)
return msg
Try doing a flow.set on defaultData and it will absolutely die a sad death. Hence the saferSerialize function plus things like uibuilder's optional <json-viewer> web component. If you have uibuilder installed, there is an example flow available for testing all the features.
I haven't, as yet, gone to the trouble of writing unserialise functions to match - that's a job for another month I'm afraid.
Anyway, back to the issues at hand. JavaScript is a complex beast and if you are getting to the point of using its Map and other features, you also need to learn about the limitations and complexities.
And, as we KEEP emphasising. You can do a direct access of a flow or other context variable ONLY IF IT IS USING THE MEMORY STORAGE library - why? Because that is NOT SERIALISING the data. Any other context storage library is almost certain to need to serialize and will fail if you pass it the wrong data.
Not entirely true but I get your point. But, this is something you have to learn about JavaScript I'm afraid. We don't have to like it but we do have to live with it - at least until someone enhances Node-RED with a better, more comprehensive object serialize/de-serialize function and wires it up to the storage libraries, debug, etc.
I'm sorry, but your idea only hides the problem, it does not solve it. That requires a lot more work.
Does that happen for memory based context?
No. Because set for memory based context is still just a JavaScript reference I believe. You can test it with this:
[{"id":"615fea9e997c4e1f","type":"debug","z":"a5f7da18de4517b5","name":"debug 38","active":true,"tosidebar":true,"console":false,"tostatus":true,"complete":"true","targetType":"full","statusVal":"","statusType":"counter","x":1095,"y":160,"wires":[],"l":false},{"id":"d98f877d7a21dd6f","type":"inject","z":"a5f7da18de4517b5","name":"Inject JSON","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"ROW1\":{\"COL1\":\"R1C1\",\"COL2\":\"R1C2\",\"odd-prop-name\":42},\"ROW2\":{\"COL1\":\"R2C1\",\"COL2\":\"R2C2\",\"i-am\":true}}","payloadType":"json","x":806.2901611328125,"y":161.2946319580078,"wires":[["b6f00c8d25c21bda"]]},{"id":"b6f00c8d25c21bda","type":"function","z":"a5f7da18de4517b5","name":"function 9","func":"const defaultData = {\n standard: {\n string: \"R1C1\",\n \"odd-prop-name\": 42,\n \"i-am\": true,\n \"date\": new Date(),\n \"array\": [1, \"a\", 2, true, 3],\n },\n functions: {\n myCustomFn: function aDifferentName() { return true },\n arrowFn: () => { return true },\n asyncFn: async function aDifferentName2() { return true },\n asyncArrowFn: async () => { return true },\n generatorFn: function* mygeny() { yield 'Hello' },\n asyncGgeneratorFn: async function* mygeny2() { yield 'Goodby' },\n },\n specials: {\n \"null\": null,\n \"undefined\": undefined,\n \"NaN\": NaN,\n \"Infinity\": Infinity,\n \"bigint\": 9007199254740992n, // Larger than Number.MAX_SAFE_INTEGER\n \"symbol\": Symbol(\"mySymbol\"),\n // @ts-ignore\n \"map\": new Map([['key', 'value'], ['key-2', 42]]),\n \"set\": new Set([1, 2, 3]),\n \"weakmap\": new WeakMap([[{ id: 2 }, 'metadata']]),\n \"weakset\": new WeakSet([{ id: 1 }, { id: 2 }]),\n \"regex\": /test/gi,\n \"error\": new Error('Process failed'),\n \"arrayBuffer\": new ArrayBuffer(8),\n \"uint8array\": new Uint8Array([10, 20, 30]), // typed array\n \"url\": new URL('https://example.come/some/path?x=this,y=that'),\n \"promise\": new Promise(() => { }),\n },\n // Circular reference example\n circular: {}\n}\ndefaultData.circular.self = defaultData\nflow.set('complexObj', defaultData) \n\nconst roundTrip = flow.get('complexObj')\nnode.warn(`typeof arrowFn?: ${typeof roundTrip.functions.arrowFn}`)\nnode.warn(`weakmap instanceof WeakMap?: ${roundTrip.specials.weakmap instanceof WeakMap}`)\n\nmsg.payload = defaultData\nreturn msg","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":980,"y":160,"wires":[["615fea9e997c4e1f"]]}]
However, if I try to save that complex object to file-based context instead, Node-RED ACTUALLY CRASHES!
Actually I'm not sure of anything now.
ChatGPT said it happens only at Disk-context, but I do not use that, and it stops at many places.
Still trying to figure out, where it becomes it.
Currently (For safety) I'm re-mapping it whenever it needed:
let fl_cegek = flow.get("cegek" ) || [];
...
for (i = 0; ...) {
let sor_ = fl_cegek[i];
let vanID = null;
if (sor_.kliensek) {
if (sor_.kliensek instanceof Map) {
node.warn(["sor_.kliensek instanceof Map", sor_.kliensek]);
vanID = sor_.kliensek.get(clientID);
}
else { // Object to a Map() again:
node.warn(["sor_.kliensek is NOT a !! Map", sor_.kliensek]);
sor_.kliensek = new Map(sor_.kliensek);
vanID = sor_.kliensek.get(clientID);
}
}
...
It is quite hard to work this out. You have to set to the file context, wait the 30sec for it to be written. And then restart Node-RED. Before testing the data.
Only when you do that will you see what has happened:
This can obviously be pretty devastating if you don't know what is going on. Because your initial testing will work just fine if you haven't restarted Node-RED. They it will fail after a restart.
My RED.util.uib.saferSerialize function tries to be rather more clear. It preserves what it can and replaces what it can't with some text:
Things like maps, sets and errors are wrapped so that you know what they are and can reconstitue them as needed:
But since @PizzaProgram is not using disc based context at all then he should not see the problem.
I don't think that is quite right. Since the data is written to disc when node-red is stopped, then it should not be necessary to wait the 30 seconds. Just restarting node-red should do it.
Ah, you are, of course, correct. 
He won't. I've tested that. I was wrong with that as well! If you test set and get in the same function node you get a very different result than if you set in one and get in another. Testing in a 2nd function node gives you the same issues as for file-based context so clearly something is happening.
However, I left a BigInt value in the memory-based test, while it doesn't crash Node-RED, it doesn't return the right value either.
Weirdly, functions DO survive. Very inconsistent.
Here is my test flow in case someone wants to do their own tests:
[{"id":"29dd9f568d0d3820","type":"group","z":"a5f7da18de4517b5","name":"File-context","style":{"label":true},"nodes":["30186b061bab84c3","3f70efc7cf2afc6a","bf6a08c4dc1d8a6c","3e144503b4cc60a8","ae61d4942367304c","731a8632240b546e"],"x":714,"y":179,"w":362,"h":142},{"id":"30186b061bab84c3","type":"debug","z":"a5f7da18de4517b5","g":"29dd9f568d0d3820","name":"debug 41","active":true,"tosidebar":true,"console":false,"tostatus":true,"complete":"true","targetType":"full","statusVal":"","statusType":"counter","x":1015,"y":220,"wires":[],"l":false},{"id":"3f70efc7cf2afc6a","type":"inject","z":"a5f7da18de4517b5","g":"29dd9f568d0d3820","name":"Inject JSON","props":[],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":775,"y":220,"wires":[["bf6a08c4dc1d8a6c"]],"l":false},{"id":"bf6a08c4dc1d8a6c","type":"function","z":"a5f7da18de4517b5","g":"29dd9f568d0d3820","name":"Set (File)","func":"const defaultData = {\n standard: {\n string: \"R1C1\",\n \"odd-prop-name\": 42,\n \"i-am\": true,\n \"date\": new Date(),\n \"array\": [1, \"a\", 2, true, 3],\n },\n functions: {\n myCustomFn: function aDifferentName() { return true },\n arrowFn: () => { return true },\n asyncFn: async function aDifferentName2() { return true },\n asyncArrowFn: async () => { return true },\n generatorFn: function* mygeny() { yield 'Hello' },\n asyncGgeneratorFn: async function* mygeny2() { yield 'Goodby' },\n },\n specials: {\n \"null\": null,\n \"undefined\": undefined,\n \"NaN\": NaN,\n \"Infinity\": Infinity,\n // \"bigint\": 9007199254740992n, // Larger than Number.MAX_SAFE_INTEGER\n \"symbol\": Symbol(\"mySymbol\"),\n // @ts-ignore\n \"map\": new Map([['key', 'value'], ['key-2', 42]]),\n \"set\": new Set([1, 2, 3]),\n \"weakmap\": new WeakMap([[{ id: 2 }, 'metadata']]),\n \"weakset\": new WeakSet([{ id: 1 }, { id: 2 }]),\n \"regex\": /test/gi,\n \"error\": new Error('Process failed'),\n \"arrayBuffer\": new ArrayBuffer(8),\n \"uint8array\": new Uint8Array([10, 20, 30]), // typed array\n \"url\": new URL('https://example.come/some/path?x=this,y=that'),\n \"promise\": new Promise(() => { }),\n },\n // Circular reference example\n circular: {}\n}\ndefaultData.circular.self = defaultData\nflow.set('complexObjFile', defaultData, 'file') \n\n// Wait for file to be written to disk\nsetTimeout(function() {\n const roundTrip = flow.get('complexObjFile', 'file')\n // node.warn(JSON.serialize(Object.keys(roundTrip)))\n node.warn(`map instanceof Map?: ${roundTrip.specials.map instanceof Map}`)\n node.warn(`set instanceof Set?: ${roundTrip.specials.set instanceof Set}`)\n node.warn(`error instanceof Error?: ${roundTrip.specials.error instanceof Error}`)\n node.warn(`arrayBuffer instanceof ArrayBuffer?: ${roundTrip.specials.arrayBuffer instanceof ArrayBuffer}`)\n node.warn(`weakmap instanceof WeakMap?: ${roundTrip.specials.weakmap instanceof WeakMap}`)\n node.warn(`typeof arrowFn?: ${typeof roundTrip.functions.arrowFn}`)\n node.send({payload: roundTrip})\n}, 36000)\n\nmsg.payload = defaultData\nreturn msg","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":880,"y":220,"wires":[["30186b061bab84c3"]]},{"id":"3e144503b4cc60a8","type":"debug","z":"a5f7da18de4517b5","g":"29dd9f568d0d3820","name":"debug 42","active":true,"tosidebar":true,"console":false,"tostatus":true,"complete":"true","targetType":"full","statusVal":"","statusType":"counter","x":1015,"y":280,"wires":[],"l":false},{"id":"ae61d4942367304c","type":"inject","z":"a5f7da18de4517b5","g":"29dd9f568d0d3820","name":"Inject JSON","props":[],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":775,"y":280,"wires":[["731a8632240b546e"]],"l":false},{"id":"731a8632240b546e","type":"function","z":"a5f7da18de4517b5","g":"29dd9f568d0d3820","name":"Get (File)","func":"\nconst roundTrip = flow.get('complexObjFile', 'file')\n// node.warn(JSON.serialize(Object.keys(roundTrip)))\nnode.warn(`map instanceof Map?: ${roundTrip.specials.map instanceof Map}`)\nnode.warn(`set instanceof Set?: ${roundTrip.specials.set instanceof Set}`)\nnode.warn(`error instanceof Error?: ${roundTrip.specials.error instanceof Error}`)\nnode.warn(`arrayBuffer instanceof ArrayBuffer?: ${roundTrip.specials.arrayBuffer instanceof ArrayBuffer}`)\nnode.warn(`weakmap instanceof WeakMap?: ${roundTrip.specials.weakmap instanceof WeakMap}`)\nnode.warn(`typeof arrowFn?: ${typeof roundTrip.functions.arrowFn}`)\n\nreturn { payload: roundTrip }","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":880,"y":280,"wires":[["3e144503b4cc60a8"]]},{"id":"df3ab33b5de9b906","type":"group","z":"a5f7da18de4517b5","name":"Memory Context","style":{"label":true},"nodes":["615fea9e997c4e1f","d98f877d7a21dd6f","b6f00c8d25c21bda","fe01c73bf63c3bd1","e16f8fc1021c9631","20e615427603c2a2"],"x":714,"y":39,"w":362,"h":142},{"id":"615fea9e997c4e1f","type":"debug","z":"a5f7da18de4517b5","g":"df3ab33b5de9b906","name":"debug 38","active":true,"tosidebar":true,"console":false,"tostatus":true,"complete":"true","targetType":"full","statusVal":"","statusType":"counter","x":1015,"y":80,"wires":[],"l":false},{"id":"d98f877d7a21dd6f","type":"inject","z":"a5f7da18de4517b5","g":"df3ab33b5de9b906","name":"Inject JSON","props":[],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":775,"y":80,"wires":[["b6f00c8d25c21bda"]],"l":false},{"id":"b6f00c8d25c21bda","type":"function","z":"a5f7da18de4517b5","g":"df3ab33b5de9b906","name":"Set (memory)","func":"const defaultData = {\n standard: {\n string: \"R1C1\",\n \"odd-prop-name\": 42,\n \"i-am\": true,\n \"date\": new Date(),\n \"array\": [1, \"a\", 2, true, 3],\n },\n functions: {\n myCustomFn: function aDifferentName() { return true },\n arrowFn: () => { return true },\n asyncFn: async function aDifferentName2() { return true },\n asyncArrowFn: async () => { return true },\n generatorFn: function* mygeny() { yield 'Hello' },\n asyncGgeneratorFn: async function* mygeny2() { yield 'Goodby' },\n },\n specials: {\n \"null\": null,\n \"undefined\": undefined,\n \"NaN\": NaN,\n \"Infinity\": Infinity,\n \"bigint\": 9007199254740992n, // Larger than Number.MAX_SAFE_INTEGER\n \"symbol\": Symbol(\"mySymbol\"),\n // @ts-ignore\n \"map\": new Map([['key', 'value'], ['key-2', 42]]),\n \"set\": new Set([1, 2, 3]),\n \"weakmap\": new WeakMap([[{ id: 2 }, 'metadata']]),\n \"weakset\": new WeakSet([{ id: 1 }, { id: 2 }]),\n \"regex\": /test/gi,\n \"error\": new Error('Process failed'),\n \"arrayBuffer\": new ArrayBuffer(8),\n \"uint8array\": new Uint8Array([10, 20, 30]), // typed array\n \"url\": new URL('https://example.come/some/path?x=this,y=that'),\n \"promise\": new Promise(() => { }),\n },\n // Circular reference example\n circular: {}\n}\ndefaultData.circular.self = defaultData\nflow.set('complexObj', defaultData)\n\n// const roundTrip = flow.get('complexObj')\n// node.warn(`weakmap instanceof WeakMap?: ${roundTrip.specials.weakmap instanceof WeakMap}`)\n// node.warn(`typeof arrowFn?: ${typeof roundTrip.functions.arrowFn}`)\n\nmsg.payload = defaultData\nreturn msg","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":900,"y":80,"wires":[["615fea9e997c4e1f"]]},{"id":"fe01c73bf63c3bd1","type":"debug","z":"a5f7da18de4517b5","g":"df3ab33b5de9b906","name":"debug 43","active":true,"tosidebar":true,"console":false,"tostatus":true,"complete":"true","targetType":"full","statusVal":"","statusType":"counter","x":1015,"y":140,"wires":[],"l":false},{"id":"e16f8fc1021c9631","type":"inject","z":"a5f7da18de4517b5","g":"df3ab33b5de9b906","name":"Inject JSON","props":[],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":775,"y":140,"wires":[["20e615427603c2a2"]],"l":false},{"id":"20e615427603c2a2","type":"function","z":"a5f7da18de4517b5","g":"df3ab33b5de9b906","name":"Get (memory)","func":"\nconst roundTrip = flow.get('complexObj')\n// node.warn(JSON.serialize(Object.keys(roundTrip)))\nnode.warn(`map instanceof Map?: ${roundTrip.specials.map instanceof Map}`)\nnode.warn(`set instanceof Set?: ${roundTrip.specials.set instanceof Set}`)\nnode.warn(`bigint instanceof BigInt?: ${Number.isSafeInteger(roundTrip.specials.bigint)}`)\nnode.warn(`error instanceof Error?: ${roundTrip.specials.error instanceof Error}`)\nnode.warn(`arrayBuffer instanceof ArrayBuffer?: ${roundTrip.specials.arrayBuffer instanceof ArrayBuffer}`)\nnode.warn(`weakmap instanceof WeakMap?: ${roundTrip.specials.weakmap instanceof WeakMap}`)\nnode.warn(`typeof arrowFn?: ${typeof roundTrip.functions.arrowFn}`)\n\nreturn { payload: roundTrip }","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":900,"y":140,"wires":[["fe01c73bf63c3bd1"]]}]
Its a result of the VM isolation.
They actually are Maps / Sets / Etc, just not matching your vm instanceof!
Alter your get function to:
node.warn(`map instanceof Map?: ${roundTrip.specials.map instanceof Map}`)
node.warn(`typename of map?: ${roundTrip.specials.map.constructor.name}`)
node.warn(`set instanceof Set?: ${roundTrip.specials.set instanceof Set}`)
node.warn(`typename of set?: ${roundTrip.specials.set.constructor.name}`)
node.warn(`bigint instanceof BigInt?: ${Number.isSafeInteger(roundTrip.specials.bigint)}`)
// Primitives do not have constructors in the same way; use typeof.
node.warn(`error instanceof Error?: ${roundTrip.specials.error instanceof Error}`)
node.warn(`typename of error?: ${roundTrip.specials.error.constructor.name}`)
node.warn(`arrayBuffer instanceof ArrayBuffer?: ${roundTrip.specials.arrayBuffer instanceof ArrayBuffer}`)
node.warn(`typename of arrayBuffer?: ${roundTrip.specials.arrayBuffer.constructor.name}`)
node.warn(`weakmap instanceof WeakMap?: ${roundTrip.specials.weakmap instanceof WeakMap}`)
node.warn(`typename of weakmap?: ${roundTrip.specials.weakmap?.constructor?.name}`)
So, in short, you can call the map/set functions as is after retrieving from context.
Is that by design I wonder or is there an issue in the code. If flow context is being serialised all the time that could have significant overheads. I think that may need a core developer to comment.
Cheers for that Steve, trying constructor names for everything except BigInt. Using > Number.MAX_SAFE_INTEGER test for BigInt
For memory-based context:
For file-based context:
(BigInt not available as it would crash Node-RED).
So, revised outcome:
- Memory-based context does retain the input data types - however, they cannot be tested the normal ways because of the use of the Node.js VM.
- File-based context breaks many data types (which is expected due to serialization) but crashes node-red on BigInt.
@Steve-Mcl - a bit of a bug around BigInt I think - it should not, of course, crash node-red. Probably needs a try/catch around the use of json-stringify-safe.
