GUNJS Database nodes - proof of concept

Hi there, I'm trying to build an app using gun.js, which is a distributed database. Gun.js is a seriously powerful and useful tool, and I'd love to be able to integrate it seamlessly in my flows.

Someone (a developer who goes by "febinjoy") has been kind enough to contribute two nodes for GUNJS, one will create a local database server, and the other is designed to allow you to write to the database.

Using the server node is fairly simple, and I got it to instantiate a DB by adding an inject node to the input and triggering that.

The client node gives you some seemingly simple config options, but no documentation on how to format the payload to get it to actually write to the DB, and the output it provides has not been useful in trying to debug my flow unfortunately.

Also, there is no way to use those nodes to read from the database, which is a real pity, and makes debugging even harder.

I was wondering if anyone else has successfully integrated gun.js into node-red using these nodes or another method. If so, I'd love some pointers! :slight_smile:

Looks like that is a dead project to be honest since the server node links to an invalid GitHub repo.

The other way to use something like this is to add the gun.js package as a require to your globals section in settings.js then you can access it from a function node.

Gun.js looks interesting and I hadn't seen it before. I'll try to find time to take a look and see if it is useful in my context.

the underlying database is being maintained though - https://github.com/amark/gun
so could be interesting.

Sorry, yes I should have made that clear. Just having a quick look. I need to find time to look at it properly as I can see that it could be super useful in a Node-RED front/back-end setup. With the Gun server hung off Node-RED. I need to see how heavy it is for small uses - clearly it is good for large cases.

In terms of footprint, here's a quote from the github repo:

Technically, GUN is a graph synchronization protocol with a lightweight embedded engine , capable of doing 20M+ API ops/sec in just ~9KB gzipped size

1 Like

#thinksmall2020 :wink:

1 Like

@dceejay :rofl:

Urm, we won't mention Angular then :wink:

1 Like

Yeah, let's not.
That gun.js does indeed look very interesting. It looks like both a good local datastore but also does replication and sync over websockets, so could have some interesting use cases both just as a node but also as a possible storage plugin.

1 Like

Not only replication and sync, but it allows you to subscribe to data changes in realtime. So if a value changes, the new value gets pushed to the subscribers immediately.

Here's my use case:
I've built a complex home automation application (webapp) and I use node-red behind the scenes to interface with all kinds of devices and services. I use MQTT to send commands from my webapp UI to node-red, from where I control the devices etc. State changes from devices and services are handled in node-red and published on MQTT topics, to which I subscribe in my webapp. For all state changes I use retained MQTT messages, so that when my app comes online and subscribes to the relevant topics, it immediately gets the latest state messages and can update the UI accordingly.

This has served me very well so far and covers a lot of use cases, however, the state data is not directly accessible, as it is abstracted away by the MQTT datastore, meaning that if I want to do anything with that data I can only do so reactively as it comes in from a subscription. If I want to work around that, I need to persist the data somewhere else, where I can get at it easily (e.g. in a context variable or some other data store), so that's creating extra work and complexity.

Replacing MQTT with Gun.js for pushing and persisting state information would kill two birds with one stone for me, as it would allow me to keep the reactive side of things very much the same, by subscribing to values in the DB. In fact I could make the subscriptions as granular as I wanted to without having to add MQTT topics to keep things separated. On top of that, it would enable me to seamlessly access any data at any time either on the webapp or in node-red without having to use a secondary datastore.

Taking it a step further, I could even consider taking MQTT out of the equation altogether, as I could create 'command objects' in gun and simply manipulate values in those objects directly from my app. On the node-red side I would be subscribed to those same objects and see the changes come in as they happen, allowing me to take action as needed to achieve the desired state, which would then be fed back into Gun and received by my app automatically. :smiley:

Supplement that with MQTT as needed... happy days! :man_dancing:

2 Likes

Certainly the idea of a subscribable content store is interesting.

I took a quick look last night but I was struggling to be able to use Gun as a simple db in Node.js rather than the browser as a starting point. All the examples are browser based & the node examples are for setting up a central server. Any ideas on that?

Setting up a Gun central server requires a custom node I believe as you need to attach it to an Express instance and I don't think you can access the Node-RED Express servers from a function node. Shouldn't be hard to do though.

Presumably that is what the existing nodes attempted to do... If they really are abandoned - maybe either the author would be willing to allow joint collaboration or they could be forked and updated.

I've done a bit of digging, and the original nodes were written for a specific use case where the author needed to mangle some data in node-red and only needed to write to the gun DB. I have managed to find his github profile and have sent him an email inviting him to join in the conversation here.
I really hope he does! :slight_smile:

Got bored on some calls so I had another go now I'm a bit fresher.

Got this example working: https://gun.eco/docs/Learn-Code#step=Step%206

[{"id":"fbf232a7.85809","type":"inject","z":"908cedac.17439","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":140,"y":340,"wires":[["96a6a8fe.76a258"]]},{"id":"96a6a8fe.76a258","type":"function","z":"908cedac.17439","name":"","func":"const Gun = global.get('Gun')\nvar gun = Gun().get('thoughts') || 'No thoughts yet'\n\ngun.set(msg.payload)\n\ngun.map().on(function(thought, id){\n    if(thought){\n        msg.payload = [thought, id]\n    } else {\n        msg.payload = ['No thought']\n    }\n})\n\nreturn msg;","outputs":1,"noerr":0,"x":270,"y":340,"wires":[["af790f1a.0ad64"]]},{"id":"af790f1a.0ad64","type":"debug","z":"908cedac.17439","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":470,"y":340,"wires":[]}]

Very simplistic but at least I can see that it does work even without a peering server defined.

Took a quick look at the code, looks like a quick proof of concept rather than anything else.

Not sure that Febin is active any more. All of his nodes were updated 2yrs 4m ago, none are linked to GitHub apart from the gun server node but that links to a GitHub that no longer exists (not even the user).

I found 3 Febin Joy's on GitHub but only 1 has any repos and those are all forks.


Hmm, some weird interaction going on here.

[{"id":"fbf232a7.85809","type":"inject","z":"908cedac.17439","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":140,"y":340,"wires":[["96a6a8fe.76a258"]]},{"id":"96a6a8fe.76a258","type":"function","z":"908cedac.17439","name":"set","func":"const Gun = global.get('Gun')\n\n// Get the \"thoughts\" set\nvar gun = Gun().get('thoughts')\n\n// Put the latest payload into the set\ngun.set(msg.payload)\n\n// Get each item in the set\n// gun.map().on(function(thought, id){\n//     if(thought){\n//         msg.payload = [thought, id]\n//     } else {\n//         msg.payload = ['No thought']\n//     }\n// })\n\nreturn msg;","outputs":1,"noerr":0,"x":270,"y":340,"wires":[["af790f1a.0ad64"]]},{"id":"af790f1a.0ad64","type":"debug","z":"908cedac.17439","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":470,"y":340,"wires":[]},{"id":"f44b41e9.2186a","type":"inject","z":"908cedac.17439","name":"","topic":"","payload":"","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":95,"y":380,"wires":[["abdf0b1c.ee0c88"]],"l":false},{"id":"abdf0b1c.ee0c88","type":"function","z":"908cedac.17439","name":"get","func":"const Gun = global.get('Gun')\nconst msg1 = {payload:[]}\n\n// Get the \"thoughts\" set\nconst gun = Gun().get('thoughts')\n\nmsg1.gun = JSON.stringify(gun) || 'NO GUN'\n\n// Get each item in the set\ngun.map().on(function(thought, id){\n    if(thought){\n        msg1.payload = [thought, id]\n    } else {\n        msg1.payload = ['No thought']\n    }\n})\n\nreturn msg1;","outputs":1,"noerr":0,"x":270,"y":380,"wires":[["53ecdc19.9d8534"]]},{"id":"53ecdc19.9d8534","type":"debug","z":"908cedac.17439","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":450,"y":380,"wires":[]}]

First run of the 2nd flow after using the 1st flow to update the value results in an empty set, running it again gives you the correct answer.

Not quite sure what is going on there but it isn't what I expected. Probably something obvious that I've overlooked.


OK, by shifting the

const Gun = require('gun')
const thoughts = Gun().get('thoughts')

To settings.js and attaching thoughts to the global variables there, it works as expected so clearly some odd interaction of the function node's VM.


Ah :bulb: gun.map().on(...) is actually an event processor. Of course, I understand better now.

Adding this to the start of settings.js:

const Gun = require('gun')
const thoughts = Gun().get('thoughts')
thoughts.map().on(function(thought, id){
    console.log('A THOUGHT OCCURED: ', thought)
})

Gets me a list of all of the thoughts listed to the console on Node-RED startup and then a new message whenever a new thought is added.

This is nice. However, it shows that you cannot realistically use Gun.js directly in function nodes. Custom nodes are required. If I could find the time, I'd have a go a writing a custom context store.

I could see a set of custom nodes that let you create a gun peer discovery server, "souls", subscribe to data updates and update data. Could then be extended with user ids as well.

Cool bit of experimentation there @TotallyInformation!

If I had the skills, I would probably structure a set of nodes as follows:

  1. A configuration node to set up the server and peers,
  2. A subscribe node, for the .on and .map methods
  3. A node for the basic read and write methods
1 Like

2 distinct use-cases here at least:

  1. A server that allows front-end's (whether data driven from Node-RED or otherwise) to sync/share data.
  2. A DB for Node-RED itself to use. Which doesn't, of course, need the server component.

Both have significant potential benefits. Especially given the ability to subscribe to changes.

1 Like

Totally agreed. :smiley:

If anyone cares to look, I've started messing with a set of nodes. Still a long way to go but you can watch for changes here:

You can install direct from GitHub but I wouldn't expect things to particularly work as yet.

3 Likes

Fantastic!! :beers: :+1:t2: :+1:t2:

I will definitely do some testing and provide feedback as soon as I have a chance!
Very much looking forward to seeing this mature, and I'm sure it will be come a popular set of nodes.

Edit 1: I have installed them, but indeed don't seem to be able to test much at this point. Not sure what input to give the set node, but whatever I try, I'm not getting any output out of either nodes.
I will keep a lookout for updates and do more testing as things progress. Feel free to PM me if you'd like me to test anything specifically.

Haha, yes I was obviously rather tired last night - sleep is a great thing and I've pushed a couple of quick updates out. Getting there slowly but still no output yet sorry. At least I have a shared reference to a "soul" now. Need to work out why my map().on() isn't working.


Right, so I've now pushed the first workable example. There are 2 nodes and a config node. The "gun get" node subscribes to changes, the "gun set" node makes changes, the config node creates the base library reference and a reference to the chosen "soul" (which is the bit that doesn't work but I now have a workaround).

Everything is likely to change so don't rely on this code but at least it is a bare-bones proof of concept.

Here is the example flow:

[{"id":"392af451.27573c","type":"gun-get","z":"908cedac.17439","soul":"dbbc7abe.617de8","x":220,"y":360,"wires":[["462a3caf.4daf54"]]},{"id":"462a3caf.4daf54","type":"debug","z":"908cedac.17439","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":490,"y":360,"wires":[]},{"id":"a883018d.a7203","type":"gun-set","z":"908cedac.17439","soul":"dbbc7abe.617de8","x":450,"y":420,"wires":[["d9652d9e.74b47"]]},{"id":"d9652d9e.74b47","type":"debug","z":"908cedac.17439","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":770,"y":420,"wires":[]},{"id":"d80d1361.42b9a","type":"inject","z":"908cedac.17439","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":200,"y":420,"wires":[["a883018d.a7203"]]},{"id":"dbbc7abe.617de8","type":"gun-config","z":"","soul":"test2"}]

Suitably ugly.


Hope you don't mind, I've changed the topic title to reflect what we are doing now. Change back if you don't like it.

2 Likes