Even a raw websocket connection uses http first and then "upgrades" to ws. Socket.IO is a lot smarter and will typically start with the http request, and use polling until the upgrade happens where it switches. So you are not waiting for the ws server to upgrade your connection.
Socket.Io does all the clever things for you and you don't want to mess with it when it is working as expected.
In any case, this is all embedded into Dashboard and Node-RED so messing is not really an option.
However, check on your server to see how loaded it is.
What OS is your server? What hardware? What else is running on it?
If the server is on a Pi, are you running a Linux desktop on the same device? Worse still, are you using that desktop to access Dashboard? If so, ditch the desktop completely if you can, it is a real drain on resources. Access from a different device.