GUNJS Database nodes - pre-release, v0.0.4

Share Your Nodes
41

So regarding the soul, this is the unique ID assigned to every entry in the DB. It is used internally by gun for references.

The .get method is used to set what gun calls the 'context', this can go as deep as needed using chained gets.

So what is now called soul in the nodes, I would call suggest calling the 'root context' or perhaps more aptly, 'root node', remember, gun is a graph DB, so the nodes and vertices terminology applies. This may be a bit confusing in the context of node-red nodes though :slightly_smiling_face:
You can have one or more root nodes in the DB, and other nodes branch off the root nodes.

1 Like
42

I can see that this node could be used as "replacement" for the current context store. global/flow/context stores could live as root contexts (as @thinkbig1979 says), it would miss internal functionality however.

gun.get('global').put({"varname":value})
gun.get('context').put({"varname":value})
gun.get('flow').put({"varname":value})

But that in turn may require the use of the additional of the "RAD API" for storage. But it is also reinventing the wheel :slight_smile:

The browser local storage part is quite cool how they sync it up with there is no connection available (but only works in the browser)

I can imagine if it would be on a more core level of node-red, like included in the settings and sync context/data to peers, it could be quite powerful. Nice stuff.

43

yes - I think it could make an interesting storage plugin. but one step at a time. (or indeed feel free to take up the challenge :slight_smile:

44

Yes, I went a bit too far with that, I'll be walking it back. It looked important when I first started reading but in practice, not so much.

And me, I'll refer to the top level as a "db" from now on. Not strictly accurate but the best approach I think for Node-RED.

Really, the GET reads 1-level of the graph and changes the chain context to it.

PUT allows you to add/change any depth of the graph starting at the current context. The object that you PUT defines the structure of the graph if properties don't yet exist or changes the entries if they do exist. At least that's my understanding so far.

SET is used in it's mathematical sense in Gun so is slightly confusing for programmers. Because SET is used in some of the simple tutorials, it looked like it was important. In truth, from what I can tell, people rarely use it in practice.

Other than the circular graph aspect, this is really the same concept as MQTT. Even the circular part makes sense when you think of it in JavaScript object terms since JS objects also allow circular references. So I think it will translate OK.

The difficulty I think we will have - that I've seen in some other cases too - is that the extreme flexibility of Gun does not translate well into the flow/node format.

What I think that would be very cool would be to be able to expose the Gun() reference to function nodes. @dceejay, is there a current, robust way of doing that? It would mean that you could do the framework/boilerplate part of managing Gun using nodes but then get to the real meat directly. Indeed, I think that the ability to expose something from a node's internals to function nodes would be massively powerful. Rather like context storage but allowing for passing of more complex objects than just those that can be serialised. Perhaps with an on/off switch in settings.js since it is such a powerful capability that not everyone might want to have it turned on - where users might abuse it in an enterprise/industrial setting for example.

And undoubtedly will be ... 2, 3, 4, ... :slight_smile:

Thanks, I did misinterpret what a soul was to begin with. The documentation for Gun is pretty bad.

As mentioned, I think that I will - at least for now - simplify the terminology and use "db" to represent the root nodes of the graph. This should be a useful concept and I've seen it used in other articles about Gun as well. Documentation can be used to explain the full concept but due to the limitations of handling root nodes, I'm going to restrict the nodes to working another level down.

What functionality are you thinking about there?

Hoping to start on that this weekend :wink: But I need to do the garden first, catch you all later. :herb: :seedling:

Please keep the ideas and thoughts coming, things are still evolving.

45

only via adding to settings.js - as per normal - nodes should/must be independent of each other - and that includes function nodes... otherwise that would allow polluting prototypes and all sorts of dodgy behaviour. so no - keep it "simple" for now.

46

well it should just implement the defined storage and/or context api... no more no less :slight_smile:

1 Like
47

SET is used in it's mathematical sense in Gun so is slightly confusing for programmers. Because SET is used in some of the simple tutorials, it looked like it was important. In truth, from what I can tell, people rarely use it in practice.

SET is used to define a common property, the value is some value created by gun internally, something like a relational database with an intermediate table to perform a more efficient join.

I used their example page

localStorage.clear();
// Read https://github.com/amark/gun/wiki/Graphs

var gun = Gun();
global = gun.get('global').put({name:'globalvar'})
local = gun.get('local').put({name:'localvar'})
flow = gun.get('flow').put({name:'flowvar'})

storage = gun.get('storage')
storage.set(global)
storage.set(local)

gun.get('storage').map().get('name').once(function(v,k){
  //console.log("name related to storage:",v)
})
gun.get('storage').once(function(v,k){
  console.log(v)
})
gun.get('global').once(function(v){
  //console.log("global:",v)
})

gun.get('flow').once(function(v){
  //console.log("flow:",v)
})

Output:

[object Object] {
  _: [object Object] {
    #: "storage",
    >: [object Object] { ... }
  },
  global: [object Object] {
    #: "global"
  },
  local: [object Object] {
    #: "local"
  }
}

flow does not have the relation with 'storage'

What functionality are you thinking about there?

Exposing it to function/change/switch node :wink:

The object that you PUT defines the structure of the graph if properties don't yet exist or changes the entries if they do exist. At least that's my understanding so far.

Yes, if the node would accept input like:

{payload:{get:val,put:{object}}}

Adds flexibility.

48

Julian after some time to think, I'd like to throw a few more thoughts your way.

Perhaps a the get node could be configured to GET a soul or GET a souls sub value? With a return type options of either a VALUE or a CHAIN (GUN object).

So you could do ....

GET("family")=>chain ➡️ GET("daughter")=>chain ➡️ GET("boyfriend")=>chain ➡️ GET("legs")=>chain ➡️ set({broken:true})

Where the family, daughter, boyfriend etc are entered in a typedinput (to permit the chaining to be fully dynamic)

Additionally, if we take the topic paradigm from MQTT where the first part is the soul and subsequent parts are the chained values?

  • GET NODE FIELDS (typed inputs)
    • get: [ msg ][ topic ]
    • separator: [ str ][ "/" ] (useful for using dot notation instead)
    • Return: [ Chain ] (return choices: value or chain)
      then a given a topic of exwife/daughter/father and a separator of / would be equivalent to gun.get('family').get('daughter').get('boyfreind')

What both versions might look like in node-red

image
image1016×431 35.6 KB

I am not certain right now which (if any) would be best but which ever way this goes, i am all about the dynamic aspect.

49

Drat! Why isn't that in the main documentation?! Well, I've edited it to change the .val references to .once. The explanation there seems to be rather different to the main docs? Or maybe I just didn't understand the docs.

Well that should be "just" creating the right wrapper for it. I was previously working on a different wrapper so should be able to adapt that.

As per Dave's note, I think that:

{
    topic: 'person/bob',
    payload:{
        name: 'bob', 
        age: 24
    }
}

will be more appropriate :wink:

This will translate to:

gun.get(msg.topic).put(msg.payload)

and here is a nice function that translates the output of GET into the more usual Node-RED msg:

UPDATE: Oops, some odd timing issues with the previous code. This is reliable but is ES6 specific:

function log(txt, value) {
    const msg = {topic:'', payload:{}}
    msg.topic = value['_']['#']
    msg.payload = Object.assign({}, value)
    delete msg.payload._
    console.log(txt, msg)
}

Example output using some of the code from the WIKI page referenced:

Hello wonderful person! :) Thanks for using GUN, please ask for help on http://chat.gun.eco if anything takes you longer than 5min to figure out!
SUBSCRIBED TO ALICE! { topic: 'person/alice', payload: { name: 'alice', age: 22 } }
Multicast on 233.255.255.255:8765
BOB! { topic: 'person/bob', payload: { name: 'bob', age: 24 } }
THE PERSON IS:  { topic: 'person/alice', payload: { name: 'alice', age: 22 } }
THE PERSON IS:  { topic: 'person/bob', payload: { name: 'bob', age: 24 } }
THE PERSON IS:  { topic: 'person/carl', payload: { name: 'carl', age: 16 } }
THE PERSON IS:  { topic: 'person/dave', payload: { name: 'dave', age: 42 } }
THE STARTUP: {
  topic: 'startup/hype',
  payload: {
    address: { '#': 'startup/hype/address' },
    name: 'hype',
    profitable: false
  }
}

Note that the startup output only shows a REFERENCE to the linked element - that's because GET only ever gets one layer of the graph. While that is fast, it won't always be helpful in the context of Node-RED.

50

OK, I get that. But the truth here is that I am not sure you could ever embody the flexibility of Gun's graph model in nodes without allowing access to code. Well, I guess you could but you would likely end up inventing an artificial language.

II could potentially create a Gun node that included a code editor, that is certainly an option though I don't like the idea because it means a lot of work and maintenance.

I can think of another workaround as well but since it perhaps doesn't quite fully stick to spirit of the principles you've reiterated, I will keep mum on that for now.

I actually don't think you will need so many nodes since you seem to be able to do things like "family/daughter/..." - as soon as I update things, that will just work.

The much bigger issue is Gun's insistence on only getting a single layer of data. That isn't too bad in code because you can add some more code to get the sub-layer(s). But for a flow, that is going to be a problem since you won't necessarily know what, if any sub-layers are needed - what if you wanted to simple say "get me all the daughter's boyfriends and all their details". Not an unreasonable request.

So I think that I will need to expand the get and once nodes to allow for recursion.

At least I have a function that will process the Gun output into something suited to Node-RED msg's, I just need to expand on that to make it recursable.

51

I think it would be useful for power users or something a bit more than simple gets+sets

Why do you say that would need a lot or work and maintenance?

I'm not certain it would?

The code entered could be stored in the flow (like say the function node, MSSQL node or unsafe function node I.e. no need for file maintenance like in uibuilder)

52

The whole point of Node-RED is NOT to write code - if you want to write code just use gun.js directly by adding to settings.js and have at it. I know it can do some nifty things but for starters a simple get/put that has subscribe capability and can auto replicate via https to remote devices sounds like a great load of function. (eg queue like capability for offline devices etc). Certainly when it comes to a storage plugin you won't need any extra to be useful.

The more similar semantics to existing capability (like MQTT) you can make it the easier it will be to use as a drop in alternative, any complication or "learning new ways of doing the same thing" and users will stay away.

1 Like
53

I fully understand that sentiment and the overall direction Dave (more now than i used to).
And I do agree the better solution would be "simple get/put" nodes.

However, if it became painfully obvious that additional desired functionality cannot be moulded into a nice node, a complimentary function node could be one way to go (much like the core function node compliments standard nodes).

PS, I was only asking Julian why he thought it'd be difficult to maintain. Phew :wink:

I do like this idea. Familiarity is good - especially when it comes to gaining traction and showing others how to use it. Suppose we just need to be careful not to be so rigid that we miss out some of the nicer aspects of Gun.

Some good discussion going on - cant wait to see where this goes. Will it be my go-to over MQTT? Will it become an integral part of and/or a killer feature of node-red? Who knows.

1 Like
54

I actually have an idea for the design of a node that would encompass most if not all of the desired functionality.

I've very crudely mocked up what the settings in a such a node may look like using the node-red-contrib-match node as a template. The match node gave me this idea, because it sequentially runs your msg.payload through the evaluations you set up and inside the node and allows you to add as many evaluations as you like.

Imagine something similar with the dropdown on the left letting you select which method (get, put etc) you want to chain on, and the field on the right allowing you to add the context name to get if you're using the get method, or the value you want to add/change or subscribe to, if that's your aim (the list I've copied into the mockup doesn't include "msg." option, but you would want it to.

The chain could be as long as you need it to be to get at whatever value in the db you want, and you wouldn't be limited to a pre-defined root context.

In the code of the node, you would basically concatenate the various user added fields into a single command to send to gun. No need for the user to write code, only to read up a bit to know what the various methods do.

What do you think?

image
image671×737 53.7 KB

55

(IMO) So long as it is possible to dynamically provide that list (e.g. can be sent in by a msg property)

Obviously the typedInputs allow using any msg property (or flow or global) but the number of entered items would be fixed by this arrangement - so a means of being fully dynamic would be required for more flexibility.

56

Have a look at the "Intro to Graphs" section of the docs on the gun.eco website, and it's corresponding JS bin playground

The example they give uses set of people linked to a company as employees (analogous to the daughter/boyfriend example)

Using the node design I suggested above, you would probably go about it something like this

In one node you would add the daughter and boyfriends to the DB

get --> daughter
put --> {name : 'Mary'}
get --> boyfriends
set --> {name : 'Tom'}
set --> {name : 'Dick'}
set --> {name : 'Harry'}

Then if you wanted to get all the boyfriends, you could use another node

get --> daughter
get --> boyfriends
map
once

If you wanted to subscribe to the ever growing list of boyfriends :wink:

get --> daughter
get --> boyfriends
map
on

57

Not sure I understand why the number of items would be fixed?
If I can just keep adding on items to the chain and use the typedInputs to tell the node what parts of my input message to use for what purpose, then what would the limitation be?

Edit: Oh wait, I understand... the query itself is set once in the node, so it assumes it will get the required input to execute that exact query every time it runs. You are talking about a scenario where basically a query is passed to the node and that query could be different every time. In that case a node could be made that simply takes a query as an input and lets gun execute that query and returns the result on it's output. How the user builds that dynamic query is up to the user...

58

Consider this. At Design Time (before deploy) - in your screen shot version - you add 4 items in the list. How would you (at runtime) query an item 5 deep or 6 deep?

You might then ask why is that needed? Well, i might need to design a dashboard that lets someone enter free text like "sales, 2019, count" or "Sales, 2019, march, total". Or perhaps you have an MQTT value with unknown depth. Or a RestAPI returns a list of parent child items that you need to query in gun - at design time you dont know the depth (so hard coding the items in a list would make it fixed)

I am not saying your design idea isn't feasible or good, just that there needs to be a means of being dynamic for more advanced automation.

EDIT

Just spotted your edit

59

Understood, see my edit on my previous post :slight_smile:

And don't worry, I did not interpret your comment as criticism, happy to see my suggestions challenged and improved upon :smiley:

1 Like
60

I've just received a message from Febin Joy, the developer who created the original gunjs nodes. He has let me know that he created those for a previous employer. He is no longer involved in that project and is unsure of its status, nor does he have access to the nodes' repo.

He asked me to share this slide deck of the architecture they used the nodes for, and said that he will definitely check in here when he has a chance, and is happy to see if he can help. :smiley:

2 Likes