I have a project where a VuMark scan triggers a timeline canvas to appear in HoloLens, using Unity 2022.3.25f1, MRTK3, and Microsoft Visual Studio 2022. The Unity project is built on UWP and operates within HoloLens. Additionally, I’m utilizing a UR10e robot on a virtual machine. I need to establish a connection between the HoloLens and the UR robot. While I’ve successfully connected to the robot from Node-RED to send messages and control it, I haven’t been able to establish communication from the robot and Node-RED back to the HoloLens. Do you think Node-RED is the correct approach for this connection, and how can I integrate it with HoloLens and Unity effectively?
I actually built something very similar a few years ago. Unity/C# has an MQTT library that you can use. That works on a publish/subscribe model and is an effective way of communicating one-way - depends on the kind of data you need to transport?
1 Like
Hi, thanks for replying. I used the provided solution and everything seems to work fine initially. However, when I built the project and ran it on HoloLens, nothing happened. The vumark scanning part didn't function, and after a few minutes, the app closed automatically without displaying any error messages. I'll try again with the link you shared and use the MQTT Client Tool - MQTTX to ensure it works correctly. Based on my data, I believe I need an RTDE connection. Do you have any knowledge or suggestions about this method?
I'm afraid not, and sounds like you'll be better off with HoloLens-specific support rather than the Node-RED side. Architecturally, it sounds like you're in the right direction
1 Like
thank you so much. I will try and share the result here.
2 Likes
Hello @joepavitt,
Thanks for your help. I tried using MQTTX, and it works perfectly. However, I encountered an issue that took some time to resolve. Since I'm using MRTK3, it's necessary to adjust the camera and add the MRTK XR Rig and MRTKInputSimulator to the scene to see everything in HoloLens. Now, I have a question about integrating this with the UR Robot. Previously, I used Node-RED for this connection, allowing me to control the UR Robot, send and receive messages, and run/stop operations. Node-RED also has MQTTX components, but I couldn't find a good resource to explain how to use them. Do you have any suggestions?
I'm not too familiar with MQTTX, but fundamentally, with MQTT in general you need two components:
- MQTT Broker - I'm guessing MQTTX provides on of these?
- MQTT Clients
In your case, Node-RED provides two features/nodes:
mqtt-in- to subscribe to any events on the broker, and take some relevant actionmqtt-out- to transmit events to the broker, for other clients to subscribe to
If your UR Robot is subscribing/listening to the MQTT Broker, then you can mqtt-out from Node-RED to control it
1 Like
MQTTX is an MQTT client. The MQTT nodes in Node-RED are MQTT Clients (as is MQTT Explorer, and many more MQTT clients)
Somewhere in your mix will be an MQTT Broker that MQTTX "talks to"
So, whatever you send to the Broker using MQTTX you can send using the MQTT-Out node. whatever you see coming into MQTTX you can also get inside Node-RED using the MQTT-In node.
there are tons of articles and videos on "Node-RED MQTT" around the internet. Example: https://www.youtube.com/watch?v=fkrbcnp8wpI
1 Like
Thank you @joepavitt and @Steve-Mcl. I could connect node-red by using MQTT out node to the unity. but still not clear how to receive the message from the UR Robot in node-red and then publishing that on HoloLens.
What communication protocols do you have available to you for the UR Robot?
Pushing to the HoloLens should be easy, as you've mentioned above you have MQTT working in your Unity application, so you can publish to MQTT, and the Unity app will be able to react accordingly.
I'm a bit confused about how to receive messages from a UR-Robot in Node-Red. Currently, I can only see the initial message I set in MQTT out, which has the same settings in the code and the Unity project. I receive this default message, but the message I'm supposed to receive from the script code in URSIM isn't being received by MQTT OUT in Node-Red.
I'm working on resolving this issue now. There are many ways to communicate with the robot, as discussed in Universal Robots forum (Communication interfaces - Robot Communication - Universal Robots Forum). For testing, I used the Dashboard server on Port 29999. However, I'm unsure if this is the correct communication method for this purpose. I think it might be better to use RTDE communication instead.
Yeah, once you're getting into the communications protocols, it's starting to get out of my comfort zone unfortunately.
Node-RED should be able to handle the lot (you may need to find some third-party nodes in the "Palette Manager" for anything not supported in core).
Find out how UR Robot is emitting content, then setup an endpoint in Node-RED for the appropriate protocol to receive them.
1 Like
Thanks, this link was really useful to unterstand MQTT in and out, Broker and Topic.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.
