Hacker News stories as trees

Share Your Projects
1

Hi There,

So something that has been interesting me for a long time is structuring HN news stories so that its visually easy to see where all the comments are... well this morning I finally had to procrastinate so much that I created a flow.

First off some details:

  • The HN Story is about autism, just randomly.
  • I use the firebase API for HN
  • For alignment I'm using the auto layout plugin from @BartButenaers
  • The automatic creating and importing of nodes into Node-RED, I using the clientcode node from my introspection package.
  • The generated nodes are from my closed-source mind-map nodes, so this won't work off the shelve. Instead the node types need to be changed in the client code node code to be something like a template node.

Also this is a complete mis-use of Node-RED flow editor, i.e., it's not a dashboard nor graphing tool.

This is the flow. The HN story id is plugged into the inject and the comments are recursively retrieved. One node per comment and one for the story. Total nodes created in this example was 625. Nodes are automatically linked according to hierarchy.

[{"id":"752ea3b8e5986228","type":"inject","z":"9edd1983093396b8","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"45438346","payloadType":"str","x":97,"y":284,"wires":[["4acf4527c64ef2c5"]]},{"id":"4acf4527c64ef2c5","type":"change","z":"9edd1983093396b8","name":"","rules":[{"t":"set","p":"storyid","pt":"msg","to":"payload","tot":"msg"},{"t":"set","p":"url","pt":"msg","to":"\"https://hacker-news.firebaseio.com/v0/item/\" & $$.storyid & \".json?print=pretty\"","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":323,"y":739,"wires":[["18c0f30926c76bc9"]]},{"id":"8846d85de1bcc165","type":"split","z":"9edd1983093396b8","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":1641,"y":737,"wires":[["4acf4527c64ef2c5"]]},{"id":"18c0f30926c76bc9","type":"http request","z":"9edd1983093396b8","name":"","method":"GET","ret":"obj","paytoqs":"ignore","url":"","persist":false,"insecureHTTPParser":false,"authType":"","senderr":false,"headers":[],"x":514,"y":479,"wires":[["98ff12c452745706"]]},{"id":"5de1ce559efcc054","type":"change","z":"9edd1983093396b8","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"children","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":1273,"y":379,"wires":[["8846d85de1bcc165"]]},{"id":"6e2bc30d1b919ba5","type":"change","z":"9edd1983093396b8","name":"","rules":[{"t":"set","p":"parentId","pt":"msg","to":"myNodeId","tot":"msg"},{"t":"set","p":"payload","pt":"msg","to":"children","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":1344,"y":710,"wires":[["8846d85de1bcc165"]]},{"id":"98ff12c452745706","type":"switch","z":"9edd1983093396b8","name":"story or comment?","property":"payload.type","propertyType":"msg","rules":[{"t":"eq","v":"story","vt":"str"},{"t":"eq","v":"comment","vt":"str"}],"checkall":"false","repair":false,"outputs":2,"x":725,"y":479,"wires":[["c7e2636763fe5761"],["5fa9e892a808ec51"]]},{"id":"c7e2636763fe5761","type":"ClientCode","z":"9edd1983093396b8","name":"prepare story node.","clientcode":"let parentNodeId = RED.nodes.id();\n\nmsg.children = msg.payload.kids;\n\nmsg.payload = [{\n    \"id\": parentNodeId,\n    \"type\": \"Blog-Post\",\n    \"name\": msg.payload.title,\n    \"info\": msg.payload.url + \"\\n\\n\" + msg.payload.text + \"\\n\\n-- https://news.ycombinator.com/user?id=\" + msg.payload.by + \"\\n\\nSource: https://news.ycombinator.com/item?id=\" + msg.payload.id,\n    \"sumPass\": false,\n    \"sumPassPrio\": 0,\n    \"sumPassNodeId\": \"\",\n    \"createdAt\": (new Date(msg.payload.time * 1000)).toISOString(), // (new Date()).toISOString(),\n    \"updatedAt\": (new Date()).toISOString(),\n    \"wires\": [\n        []\n    ],\n    \"icon\": \"@gorenje/node-red-mindmap/hn.svg\",\n    \"l\": false\n}]\n\nmsg.parentId = parentNodeId;\n\nnode.send(msg)\n","format":"javascript","x":1013,"y":379,"wires":[["0636d62671a4a782","5de1ce559efcc054"]]},{"id":"ce597d517d3af8fd","type":"ClientCode","z":"9edd1983093396b8","name":"link parent to child","clientcode":"let source = RED.nodes.node(msg.parentId)\nlet target = RED.nodes.node(msg.myNodeId)\n\nRED.nodes.addLink({ source: source, sourcePort: 0, target: target });\nRED.view.select([target, source]);\n\nnode.send(msg)\n","format":"javascript","x":1015,"y":534.5,"wires":[["03302be69c458330","6e2bc30d1b919ba5"]]},{"id":"5fa9e892a808ec51","type":"switch","z":"9edd1983093396b8","name":"not dead","property":"payload.dead","propertyType":"msg","rules":[{"t":"false"},{"t":"null"}],"checkall":"true","repair":false,"outputs":2,"x":725,"y":533,"wires":[["b53467c717790ab8"],["b53467c717790ab8"]]},{"id":"0636d62671a4a782","type":"ClientCode","z":"9edd1983093396b8","name":"import nodes","clientcode":"/*\nlet nde = [\n    {\n        \"id\": RED.nodes.id(),\n        \"type\": \"Bookmark\",\n        \"name\": \"Bookmark\",\n        \"info\": \"https://hacker-news.firebaseio.com/v0/item/45438346.json?print=pretty\\n\\n\",\n        \"sumPass\": false,\n        \"sumPassPrio\": 0,\n        \"sumPassNodeId\": \"\",\n        \"createdAt\": \"2025-10-02T07:00:08.553Z\",\n        \"updatedAt\": \"2025-10-02T07:00:08.553Z\",\n        \"x\": 588,\n        \"y\": 247,\n        \"wires\": [\n            []\n        ],\n        \"icon\": \"@gorenje/node-red-mindmap/hn.svg\"\n    }\n]\n*/\n\nlet nde = msg.payload;\n\nRED.view.importNodes(Array.isArray(nde) ? nde : [nde])\n\nnode.send(msg)\n","format":"javascript","x":1245,"y":302,"wires":[[]]},{"id":"2a677cc320235746","type":"ClientCode","z":"9edd1983093396b8","name":"import nodes","clientcode":"let nde = msg.payload;\n\nRED.view.importNodes(Array.isArray(nde) ? nde : [nde])\n\nnode.send(msg)\n","format":"javascript","x":1015,"y":587.75,"wires":[["ce597d517d3af8fd"]]},{"id":"03302be69c458330","type":"debug","z":"9edd1983093396b8","name":"debug 346","active":true,"tosidebar":false,"console":false,"tostatus":true,"complete":"true","targetType":"full","statusVal":"","statusType":"counter","x":1293,"y":534.5,"wires":[]},{"id":"b53467c717790ab8","type":"switch","z":"9edd1983093396b8","name":"not deleted","property":"payload.deleted","propertyType":"msg","rules":[{"t":"false"},{"t":"null"}],"checkall":"true","repair":false,"outputs":2,"x":725,"y":587,"wires":[["5a16e03d1724667c"],["5a16e03d1724667c"]]},{"id":"b32ed12fe4aee645","type":"ClientCode","z":"9edd1983093396b8","name":"prepare comment node.","clientcode":"\nlet myNodeId = RED.nodes.id();\n\nmsg.children = msg.payload.kids;\n\nmsg.payload = [{\n    \"id\": myNodeId,\n    \"type\": \"Comment\",\n    \"name\": msg.payload.text.substr(0, 30) + (msg.payload.text.length > 30 ? \"...\" : \"\"),\n    \"info\": msg.payload.text + \"\\n\\n-- https://news.ycombinator.com/user?id=\" + msg.payload.by + \"\\n\\nSource: https://news.ycombinator.com/item?id=\" + msg.payload.id,\n    \"sumPass\": false,\n    \"sumPassPrio\": 0,\n    \"sumPassNodeId\": \"\",\n    \"createdAt\": (new Date(msg.payload.time * 1000)).toISOString(), // (new Date()).toISOString(),\n    \"updatedAt\": (new Date()).toISOString(),\n    \"wires\": [\n        []\n    ],\n    \"icon\": \"@gorenje/node-red-mindmap/hn.svg\",\n    \"l\": false\n}]\n\nmsg.myNodeId = myNodeId;\n\nnode.send(msg)\n","format":"javascript","x":1015,"y":641,"wires":[["2a677cc320235746"]]},{"id":"5a16e03d1724667c","type":"switch","z":"9edd1983093396b8","name":"not flagged","property":"payload.text","propertyType":"msg","rules":[{"t":"neq","v":"[flagged]","vt":"str"}],"checkall":"true","repair":false,"outputs":1,"x":725,"y":641,"wires":[["b32ed12fe4aee645"]]}]

Doing the importing, nodes are created without labels for compactness. The import took a minute and half, longer than the gif:

hn-import-2

Once the nodes are imported, I can alignment them - automagically - into a tree structure as comments on the story, followed by the comments on comments, etc. Alignment algorithm was "ELKjs Layered Downward":

hn-import-part2

Also shown is that the content of comments and links to the comments are stored in the node and can be viewed in the info panel. So I can also navigate through the structure reading those comments of interest.

Why?

Because I wanted to include this in my mindmap. So that when I drag a link to a HN story into my mindmap, it would also create all the comments and subcomments of the story. That step isn't shown here.

3 Likes
2

So it's now integrated into my mindmap tool:

hn-impoert-oom

All I do is drag a HN url onto Node-RED and it does the rest. The main story is imported and all comments and sub-comments are automagically imported as nodes (i.e. I can actually execute this stuff as a flow, sending a message through it).

The nice thing is that all these nodes are then searchable using the Node-RED search. I can link them to existing nodes. I can navigate them as any other nodes in Node-RED, i.e., cursor navigation over flows.

Auto-layout is now right to left, that makes a big difference. Unfortunately I still have to do that manually because I haven't yet worked out when all nodes have been imported - doing this recursively in parallel doesn't help...

3 Likes
3

I love how you hack the hell out of node-red :')

3 Likes
4

He is the most creative user!

2 Likes
5

I won't be doing any of this if @knolleary and @dceejay hadn't done such a superb job of creating clear and understandable APIs for the frontend - all respect to both of them :glowing_star:

Eg. linking two nodes - 4 lines of code:

let source = RED.nodes.node(msg.parentId)
let target = RED.nodes.node(msg.myNodeId)

if ( source && target ) {
    RED.nodes.addLink({ source: source, sourcePort: 0, target: target });
    RED.view.select([target, source]); // make link visible in workspace
}

Creating a node and installing it on the workspace - 1 line of code:

let nde = { // this definition can be found in the export dialog
        "id": RED.nodes.id(),
        "type": "template",
        "name": "",
        "field": "payload",
        "fieldType": "msg",
        "format": "handlebars",
        "syntax": "mustache",
        "template": "Add content here",
        "output": "str",
}

RED.view.importNodes(Array.isArray(nde) ? nde : [nde])

In lot of ways, Node-RED is the best open source tree graph manipulation tool that can also execute code :wink:

All respect to @BartButenaers who is way more creative than I am: blockly support, terminal support, interactive SVG graphics plus many more.

I took a lot of inspiration from Bart and he gave me a lot of support in the beginning of my NR career - big :heart: out to Bart!

In the end: there are many that came before me and there will be many who come after me.

3 Likes
6

I agree with it. The APIs are really easy to use. They did an amazing job. However, there is still room for improvement and that is what I'm trying to achieve. For example, I'm creating a Proxy that automatically let us get the related node instance doing this.referenceNode instead of RED.nodes.get(this.referencedNode). During the creation of nodes I change all config props that are node references (using the node's schema definition) by a proxy. When I'm done with this, I will try to apply the same idea to with typedInputs.

I hope I can reach your level with the idea Im cooking.

7

I've been silently observing what you're up to. I think one thing that you have to remember are the base principles of Flow Based Programming and the assumptions made in Node-RED because of that. For example, FBP stipulates that nodes are independent of one another, so Node-RED has zero support for dependencies on node initialisations, node references to one another (i.e. navigating the flow from one node) and that won't change.

I've seen it come up several times here in the forum and there won't be - as far as I would think - changes to those core assumption of Node-RED.

That same with message passing and the immutability of messages, i.e. a node can't change the message of another node.

That is not to say it's impossible to work around those restrictions, it's just that Node-RED core will never support anything that breaks the core assumptions.

One of my first nodes where Artificial Neural Network nodes. Those nodes required knowledge of the network in which they were embedded. It wasn't easy to implement, but I did get it working ... won't do it again though.

1 Like
8

I've now updated the serverless Node-RED (i.e.. everything runs in the browser) to support this flow, so if anyone wants to import their own HN news threads, change the payload value in the serverless Node-RED to the story id.

3 Likes