GUNJS Database nodes - pre-release, v0.0.4

[TotallyInformation]

Just pushed a small update with a stand-alone test file containing some examples of Gun.js based on the tutorials but messing about trying to work out how to be able to retrieve >1 level of hierarchy. That isn't working yet because I haven't worked out the async logic needed, standard recursion doesn't work. I think this logic will be needed to make full use of Gun in Node-RED. If anyone can work out the logic, please do!

Thanks for everyone's feedback so far, it has been really helpful.

@thinkbig1979

Then if you wanted to get all the boyfriends, you could use another node

The point here is that you wouldn't always know the shape of the data. All of the Gun.js examples seem to assume that you do know the shape and therefore you write code to go get what you want. That isn't a reasonable assumption to me. I don't want to always have to write very specific flows/code to get back a set of data, sometimes I just need a "tell me everything" option. This would be particularly true if trying to use Gun as a context store. The alternative is to use JSON.stringify to wrap the data into a string but that doesn't feel quite right to me.

@thinkbig1979

I actually have an idea for the design of a node that would encompass most if not all of the desired functionality.

Thanks for that. I had thought about something of that nature but again haven't really had the time to explore it. But the idea is certainly worth exploring.

@Steve-Mcl

So long as it is possible to dynamically provide that list

Agreed. However, I suspect that would in in a v2 rather than an initial release

---

I think that my next stage will be to consolidate the current nodes & make sure they work correctly with slash delimited souls (e.g. "daughter/boyfriends") with the single-level hierarchy.

As that probably constitutes a minimum viable product, I'll probably publish that as a v0.1 release.

After that I'll likely look at the "match" node idea.

Sadly, I don't know that the context store is going to go anywhere until I can get a multi-depth hierarchy get working.

I do still have an idea about how settings.js global entries could be interfaced to the custom nodes in order to be able to easily get references to the objects defined in the nodes. It would certainly fall under advanced use case but would, I believe, be robust. But for now, I'm going to just keep that as an idea. Hopefully, we can find better solutions.

@dceejay, I do get the desire to ensure that the core principles of Node-RED are followed but I think we would all agree that Node-RED is not realistically a "no code" framework for advanced cases. But rather a "low code" framework. However, I do want to focus on the simple parts first so that non-coders can leverage gun.js - I'm just not too sure how much use it will be - no reason not to try though

@Steve-Mcl

Why do you say that would need a lot or work and maintenance?

I'm only going by how much work it was to get an editor working. Perhaps I'm wrong. My concern is that whenever internal changes to NR are made, changes to the gun-function node would also need to be made - but truth be told, I'm not sure how often the function feature would actually change. Actually, I don't even know how many lines of code the function node is?

I think that a gun-function node would be in v3 if at all. With @thinkbig1979's idea, it may not be needed. Let's see. Of course, if you wanted to contribute a node to the repo, it would be gratefully received.

[Steve-Mcl]

@TotallyInformation

Of course, if you wanted to contribute a node to the repo, it would be gratefully received

I'm watching your repo for latest changes, then I'll update my fork & see your direction and how I can contribute.

So if there is anything you think you might need a pair of hands on, let me know.

[thinkbig1979]

@TotallyInformation

sometimes I just need a "tell me everything" option.

If you know the root context, let's say it's daughter, you can do this:
gun.get('daughter').map().once(function(data){
console.log( data);
});

That will return everything inside the daughter context.
The same is true at every level down. So start at the context depth you know/want and grab everything underneath...

If you go to the JSbin example I linked to before, comment out the last query and add this one to see this in action. Note that startup/hype is actually the root context in this example (there is no startup context).

gun.get('startup/hype').map().once(function(data){
  console.log(data);
});

Does that help?

[cinhcet]

have you looked into the time feature of GUN?
Judging from the docs, it does not seem very useful for timeseries data

@TotallyInformation it is already featured on https://github.com/amark/gun/wiki/Awesome-GUN

[TotallyInformation]

@thinkbig1979

Does that help?

Not really.

The point is that the on/once only gets a single level. To get everything, you have to recurse down the tree but because Gun is a streaming server, the returned data is asynchronous. If Gun was truly promise-based, this wouldn't be that much of an issue since you could just await everything and only do the final output once everything was all done. However, the then and promise features are marked as not official and "buggy". Doing it all with callbacks is beyond my current skills. It will have to wait.

@cinhcet

@TotallyInformation it is already featured on https://github.com/amark/gun/wiki/Awesome-GUN

Hmm, I think one could say that they'd "jumped the Gun"!

@cinhcet

have you looked into the time feature of GUN?

Not a chance! I'm confused enough as it is!

[TotallyInformation]

OK, that's it for tonight. I'm starting to get annoyed again because every time I think I've cracked it, it appears that its cracked me instead.

I've pushed a new version out - v0.0.1-dev6

• A tab of examples that you can import via Library>Import
• Refactored the code, a lot neater and more consistent
• Combined PUT and SET nodes to a single node

Sorry, if you already have some flows using previous versions, I recommend that you scrap them before loading the new version - mainly because of the combined put/set node which is also renamed.

So what isn't working? Well, check out the examples - trying to use slash-delimited puts and gets doesn't seem to be doing what I expect and at the moment, I can't work out whether that is my code or the way that Gun works.

If you get a chance, please do take a look, I'm sure there are plenty of better (and younger!) brains than mine that will spot the problems.

@Steve-Mcl

So if there is anything you think you might need a pair of hands on, let me know.

I think that, until we can bottom out this issue, probably no point in doing much else I'm afraid. So that is the focus for now.

Catch you all tomorrow I expect.

[cinhcet]

@TotallyInformation

The point is that the on/once only gets a single level. To get everything, you have to recurse down the tree but because Gun is a streaming server, the returned data is asynchronous. If Gun was truly promise-based, this wouldn't be that much of an issue since you could just await everything and only do the final output once everything was all done. However, the then and promise features are marked as not official and "buggy". Doing it all with callbacks is beyond my current skills. It will have to wait.

recursing down a tree with async callbacks can be realized neatly with Promise.all(), even if the rest of the API is not promise based, since you can always return a Promise from the recursive calls.

[RNCTX]

Not to derail, but are you guys familiar with Postgrest?

Rather than trying to reinvent this wheel, what makes a lot more sense to me in cases like these is to let the DB do what the DB is good at. They've only been working on performance, fine-grained access control, and handling every data type imaginable over there for the past 2-3 decades, after all ;).

If you want non-relational data go ahead, it works with JSONB tables too.

http://postgrest.org/en/v7.0.0/

The "mydb" contrib node for Postgres is actively maintained and works fine with complex JSONB from my experience with it if you need complex JSON inserts.

@open-kappa/node-red-contrib-mydb

There are six different existing methods / plugins for pub/sub with Postgrest depending on your preference (websockets, MQ server variants, and AWS services like SNS).

[BartButenaers]

@RNCTX

Rather than trying to reinvent this wheel

Personally I don't see a problem in Node-RED supporting multiple technologies. And not sure whether the Postgress node supports all Gun features, like e.g. having a change listener... Anyway this topic is about the development of the Gun nodes, so I would suggest to discuss about this in a separate topic.

[thinkbig1979]

@RNCTX
I think Postgress is a great database, and has a lot of valid use cases in and outside of node-red, but it's not native to the JavaScript environment, it's not a graph database, it's not offline first or natively distributed, and it does not natively support reactive programming.

I think those are the features that make Gun so interesting in this context.

[TotallyInformation]

@RNCTX

Postgrest

many thanks for the reference. While interesting, the really key thing for me about Gun is that it is offline-first and peer-to-peer. So it doesn't require a server at all. Something that should be very useful for lightweight environments. Single-page applications could also use it of course and it is trivial to connect to a back-end instance configured under node-red - no additional service required.

Of course, while all of these things might endear it to Node-RED's typical user base, it will be of little use if the terminology and actual use of Gun is incomprehensible to the majority of users. That is my worry at the moment. I definitely seem to be missing something key because things aren't working as expected.

[TotallyInformation]

@cinhcet

recursing down a tree with async callbacks can be realized neatly with Promise.all(), even if the rest of the API is not promise based, since you can always return a Promise from the recursive calls.

I'm sure it can. Unfortunately, I am not a full-time JavaScript developer and therefore I lack the skills. Example code would be most welcome

I have now discovered that Gun does have a couple of functions that get everything and I will build those into the current nodes as an option.

[cinhcet]

@TotallyInformation

Example code would be most welcome

According to your post, it seems that it is obsolete now, but here you go:

  function getAllChildrenRecursively(obj) {
    return new Promise(function(resolve, reject) {
      asyncCallToSomething(obj, function(result) {
        let itemList = [];        
        if(result.hasOwnProperty('items')) {
          itemList.push(...result.items);
        }
        if(result.hasOwnProperty('children')) {
          Promise.all(result.children.map(function(child) {
            return getAllChildrenRecursively(child)
          }))
          .then(function(temp) {
            temp.forEach(function(t) {
              itemList.push(...t);  
            });
            resolve(itemList);
          });
        } else {
          resolve(itemList);
        }
      });
    });
  }

Disclaimer: This has nothing to do with GUNJS directly, but it is a general example how one can do an async recursion. It assumes that asyncCallToSomething has a callback where the argument is an object with an items and children property. It returns all items in a list. Also note that I have not tested that code, but am pretty sure it should work, except for syntax errors.
Explanation: The return of the async call is checked for an items property and all items are added to the itemList. If the returned object still has children, it will run the same function recursively for each children. The magic now is the Promise.all, which is a promise that only resolves if all promises that are given in its argument are resolved, i.e. it waits until all async calls are finished (they are still executed asynchronously). Then all items that are returned from each recursive call are added to the list.

I would have given an example for GUNJS, however, I don't really get if there is something like a hasChildren method or if my example even make sense in the context of that database

[BartButenaers]

@dceejay

but also as a possible storage plugin

Hey Dave,
Perhaps a stupid question. I assume that Gun as a storage plugin doesn't expose all the features that Gun offers? I assume that it will have the same possibility compared to the existing local-filesystem storage plugin, which simply persists the global/flow data in a file? Perhaps Gun might this this more efficient (e.g. in memory caching, ...). Or are there other advantages to create such a storage plugin for Gun?

[dceejay]

Well - basically yes - from the Node-RED side it would just be set and get etc... but it would open up the possibility for on/offline capability - auto syncing back to a mothership - so auto queuing of context updates.
That could also be one possible way to do a deploy to raft of devices ? deploy local - replicate to the others.
And potentially if you had several flows in parallel (horizontal scaling) then you could use the subscribe capability to bust the local cache so any reads get the correct data etc...
Just thinking out loud.(and yes there may need to be some other signalling involved... but)

[TotallyInformation]

@cinhcet

According to your post, it seems that it is obsolete now, but here you go:

That's great, many thanks. I will study it until I understand it (may take a while)

@cinhcet

The magic now is the Promise.all , which is a promise that only resolves if all promises that are given in its argument are resolved, i.e. it waits until all async calls are finished (they are still executed asynchronously). Then all items that are returned from each recursive call are added to the list.

Argh! I would never have worked that out.

@cinhcet

I don't really get if there is something like a hasChildren method

Not directly but each returned level contains links to any lower-level data which can simply be passed to another get. So same idea.

@cinhcet

if my example even make sense in the context of that database

Yes, I think so, I'll need to try it out.

@cinhcet

seems that it is obsolete now

Not so sure. Looking at their version, they claim it can have timing issues which implies to me that they are not doing it how you are but rather waiting a set time and assuming everything has finished. While that might be useful for some cases, not so much in others.

[TotallyInformation]

@dceejay

use the subscribe capability

Certainly the sync, on/off-line capability is intriguing but it is the subscribe feature that I personally find most appealing. Not sure how it would be surfaced but the ability to subscribe to context variable changes would be massive.

[cinhcet]

If you have any questions, please let me know!

I can remember when I first needed something like that it also took me a while to get it right

[thinkbig1979]

@TotallyInformation

The point is that the on/once only gets a single level

I was really confused there for a bit, as the example I showed clearly returned multiple levels of data. What I didn't realize is that it does not return all the data at once, but one level at a time. Sorry for not spotting that earlier.

The gun.load method may be what you are looking for:

GUN — the database for freedom fighters - Docs v2.0

Here's a snap of the object that was returned when I ran the following query in the JSbin example...

//don't forget you need to add the library separately, as this method is not in the main library
gun.get('startup/hype').load(function(data){
  console.log(data);
});

Looks promising!

[TotallyInformation]

Hi all, just to let you know that I found the problems in my code and the gets are working great now.

Just adding a flag to get the full hierarchy as an option so expect an updated version later this evening.

Assuming you guys don't tear it apart, I think that might then become v0.1.0 and be published to npm because I think it is actually usable if still basic.