Socket.io, migrating from 2.x to 3.0

kevinGodell · 28 November 2020 16:57

I see that socket io has released version 3 on November 4th and it has breaking changes. It also states that a v2 client cannot connect to a v3 server, and vice versa.

Has there been any discussion or plans on upgrading to the newer version?

I just found out about this new version a moment ago and it seems my nodes can be changed rather easily to adapt to the breaking changes. I suspect that having 2 different socket io dependencies of version 2 and 3 on the same server and client will probably not play nicely with each other, but I have done no testing yet.

TotallyInformation · 28 November 2020 18:08

Thanks for raising this. I'll need to check it out for uibuilder as well.

In theory, npm should take care of multiple module versions so we will have to see.

kevinGodell · 29 November 2020 04:37

Perhaps my eyes are deceiving me, or maybe I am too tired to properly read the output of htop, but I am getting really low cpu usage now when streaming video content using socket.io@3. I had to go to another computer and open up more browsers to stream more video simultaneously to stress the little pi to see if I could get the cpu to spike with all of the socket.io activity. Normally, with that large amount of video playing, I would see the node-red process hover around 40% or more due to socket.io. As of now, it stayed around 7% at peak load.

BartButenaers · 29 November 2020 08:52

Morning Kevin,
Are you using socket.io outside of Node-RED, or have you fixed (locally) the Node-RED dashboard to use the latest socket.iocallyo version?
Bart

kevinGodell · 29 November 2020 14:33

I forked the node-red-dashboard, updated the socket io dependency to v3, ran grunt, and then installed it in place of the original node-red-dashboard. I needed it updated so that it can provide socket io v3 client code. It seemed to run fine without needing any code changed since it was not using any of the removed or changed features of v2.

Then I fixed my broken code in node-red-contrib-mp4frag with this commit and node-red-contrib-ui-mp4frag with this commit. The client side fix was very small, but the server side required the removal of the .binary() methods.

I would suspect that most people would not notice a big cpu load change unless they are streaming large amounts of binary data, such as video.

In my implementation, I create a socket io server, attach it to the existing http server, and create a namespace for each camera. So, 1 server per class, and 1 namespace per instance.

BartButenaers · 29 November 2020 19:10

So it is a breaking change, but seems like a rather small change to me. And I'm doubting that the current UI nodes would be impacted by this change? Or did I misinterpret anything perhaps?

kevinGodell · 29 November 2020 19:15

You are correct. There are breaking changes, but none that I have seen that affect node-red-dashboard.

dceejay · 29 November 2020 19:56

Apart from the dropping of node.8 - which although now beyond end of life the core of Node-RED still supports...for now. Glad to know that the migration path is easy though so yes any hints or advice greatly appreciated.

Happy to have a PR to show us the way (but won't merge it for a while).

kevinGodell · 30 November 2020 13:31

I am no expert when it comes to semantic versioning and npm hackery, but I am thinking about a ranged peer dependency, if possible. If socket io is listed as a peer dependency, then can the user pick the version for node-red-dashboard to use?

I am not even sure if that is possible. Frankly, I just read about peer dependencies and I still don't get it. I will try some variations to see what can work. Maybe something like this?

"peerDependencies": {
        "socket.io": "^2.3.0 || ^3.0.3"
    },

dceejay · 30 November 2020 14:26

Hmm no that's even worse as then the server somehow has to handle both at the same time - I have no idea how the server could load both libraries concurrently without a heap of complication - and because they explicitly say 2.x client can't connect to 3.x servers etc then it'll all end badly very quickly.

Either you can create nodes that require 3.x and would thus only work with your fork of the server - and wait until we move and drop 2.x (March/April 21 - but maybe betas before then of course) - or you create 2.x versions that work with what we have today knowing you can bump forwards as soon as... - or you maintain both in parallel (one on say a 2.x.y branch and one called 3.x.y) so users can get both fairly easily. Or we look at dropping socket.io altogether (as indeed we aren't using many of it's features) - and just move to raw ws. (A not insignificant effort but...)

kevinGodell · 30 November 2020 15:05

I guess I misunderstood. I thought socket io would have to be installed next to node-red-dashboard as opposed to being under it, allowing only a single version to be installed on the server.

I changed my nodes to work with either v2 or v3 and they will use whatever version comes with node-red-dashboard. On the server side, socket io now stores some things as a Map and I do a simple check before doing the cleanup here and here.

Yes, that would be a major task. I had wanted to also implement raw ws, but socket io spoiled me.

dceejay · 30 November 2020 15:16

Ah good - if you can handle either by just detecting which we provide then great - that would be best. Yes no need for declaring dependencies your side as we already have it - you would only need to if we don't. And peer dependencies especially just flag warnings and annoy people and don't actually install anything or do anything really useful. (imho)

kevinGodell · 30 November 2020 17:09

I just tested some scenarios for node-red-contrib-mp4frag using socket io as a dependency. I am trying to prevent my server side node from being tightly coupled to node-red-dashboard because it could still be used as a media server outside of the dashboard. It seems that setting the peerDependency in my node for socket io satisfies the requirement if I use the published dashboard depending on v2 or my fork of it using v3.

TotallyInformation · 30 November 2020 17:29

Why not just set it as a normal dependency? Surely that will work, npm should sort things out. If you specify the dependency as ^2 then it will use an existing entry if it can or install v2 if it can't, similarly with v3.

dceejay · 30 November 2020 17:54

@TotallyInformation I could swear you are the one who keeps telling me the dashboard is too heavy as it loads lots of libraries Having it load socketio 2 and v3 hs to be a waste of space.

kevinGodell · 30 November 2020 17:55

Since my node is designed to work with v2 or v3, it will happily use whatever it already finds installed, or give an error on the console stating that the peer dependency was not found. Since most regular users will use node-red-dashboard, or some more advanced users will use node-red-contrib-uibuilder, or super duper high techies will go headless, socket io will be available somehow.

TotallyInformation · 30 November 2020 21:02

Who moi?

But I don't think the Dashboard will do that will it? Node-RED's Dashboard nodes will load whatever its package.json calls for and any Dashboard extension nodes will need to match. But Dashboard isn't the only thing using Socket.IO. So a non-Dashboard node could serve up a different version if it needed to couldn't it?

Yes, that makes Node-RED that much heavier I suppose but honestly, Node-RED as a whole is already pretty large and it is only temporary until all nodes catch up. Surely this is inevitable since it is very unlikely that we will be able to update Dashboard, uibuilder and anything else that uses Socket.IO at the same time.

All I was saying is that while the client and server for a particular use-case have to match, I don't think Node-RED as a whole does & I think that npm will take care of it. But I'll admit to not being totally sure. I guess we will find out pretty soon.

kevinGodell · 6 December 2020 18:55

A nice little feature of v3 that I was just playing with is passing an auth object from the client to server.

On the server side, socket.handshake.auth is populated with data that is passed from the client side that added auth: { key: 'somekey'}. In v2, if you wanted to pass some type of key, it had to be added as query: { key: 'somekey'} which puts that data into the url, making it very visible, or you will have to add your own authentication event to pass the key more securely. It is definitely a nice little addition.