RosBridgeClient for UWP

[arunavanag591]

Hello,

I would like to send an array of float values as a std_msgs/String. However, I could not locate many examples for publishing message from Unity to ROS except for CameraImagePublisher.cs. Is it possible using ros-sharp?

My message would look something like this: "[718.23, 234.21, 324.54 . . . . .. ]"

Also, after importing ROS-sharp as given here. The menu items do come, but when I am building it for UWP,

the following errors are thrown:

Assets\RosSharp\Scripts\JointStateSubscriber.cs(21,30): error CS0246: The type or namespace name 'RosSocket' could not be found (are you missing a using directive or an assembly reference?)

Assets\RosSharp\Scripts\JointStateSubscriber.cs(45,40): error CS0246: The type or namespace name 'Message' could not be found (are you missing a using directive or an assembly reference?)

Assets\RosSharp\Scripts\OdometrySubscriber.cs(47,44): error CS0246: The type or namespace name 'NavigationOdometry' could not be found (are you missing a using directive or an assembly reference?)

I don't know why these errors are thrown as in the plugins the ROSBridgeClient.dll is already imported. I am building it as a C# project, to deploy onto my mixed reality device.

[MartinBischoff]

Hi @arunavanag591 !

Concerning your 1st question:

It is absolutely possible to use ROS# for publishing std_msg/String topics from within Unity. Here is an example:

using UnityEngine;

namespace RosSharp.RosBridgeClient
{
    [RequireComponent(typeof(RosConnector))]
    public class StringMessagePublisher : MonoBehaviour
    {
        private RosSocket rosSocket;
        public string topic = "/message";        
        private StandardString Message;
        private int publicationId;       

        void Start()
        {
            rosSocket = GetComponent<RosConnector>().RosSocket;
            publicationId = rosSocket.Advertize(topic, "std_msgs/String");
            Message = new StandardString();
        }

        public void Update()
        {
            Vector3 values = new Vector3(Random.Range(0, 1000.0f), Random.Range(0, 1000.0f), Random.Range(0, 1000.0f));
            Message.data = values.ToString();
            rosSocket.Publish(publicationId, Message);
        }
    }
}

Just drop the above StringMessagePublisher into some GameObject of your Scene together with a RosConnector.

Here is another code example how to publish std_msg/String messages with ROS# from .NET without necessarily using Unity.

Yo find all ROS message types currently supported by ROS# in Message.cs.

Concerning your 2nd question:
I guess the problem you are having here is that the websocket-sharp library we are using does not support UWP. We are discussing that problem in issue #21. I'm going to write an update there on things I found out just now triggered by your question.

Let us continue discussing your 2nd question in issue #21.

If you are fine with the answer to your 1st question please close this issue.

Best,
Martin

[arunavanag591]

Hi @MartinBischoff though the answer look satisfying, however, I haven't been able to use ROS-Sharp due to issue #21

I tried a bunch of workarounds, but its only the issue with the ROSBridgeClient.dll which throws the error.
I have used websocket-sharp.dll in UWP, I did not have issues. When I had to declare socket and other related directives, all I had to is

 #If UNITY_EDITOR 

#endif

One such example you can find here

[MartinBischoff]

@blommers provides a NetStandard 2.0 Version of the UrdfImporter. Have you tried this one?

Instead of using the RosBridgeClient.dll in a Plugins folder, you can also use the source code of the RosBridgeClient project directly in Unity and add any required preprocessor directives there.

[arunavanag591]

When I do that I have issues with the ROSConnector and other scripts

Assets/RosSocket.cs(201,45): error CS1644: Feature **null propagating operator** cannot be used because it is not part of the C# 4.0 language specification

If I change the target to 4.5 then websocketsharp fails.

[arunavanag591]

Hi, @MartinBischoff could you find a solution this? I have been putting a lot of time on this, however nothing yet.

[MartinBischoff]

Hi @arunavanag591 would you please report what you have tried out thus far? What worked and what didn't? The info you provide will be helpful for others trying to solve your problem.

In our group there is no demand for UWP at the moment. Maybe you want to ask others for support? @jaguar243 @Jasonthefirst

[arunavanag591]

Hi,
So I have tried including the Rosbridge lib directly as an imported package, and as I mentioned earlier

When I do that I have issues with the ROSConnector and other scripts

Assets/RosSocket.cs(201,45): error CS1644: Feature null propagating operator cannot be used because it is not part of the C# 4.0 language specification

it throws me such error. I tried setting .NET target to 4 it solves that issue however it doesn't compile websocketsharp. I downloaded the websocket sharp library and created my own dll, but it did not help either.

I tried using

#if WINDOWS_UWP

#endif 

It still fails to build both the rosbridge lib and websocket dll

[rwarrier243]

Hi, I came across this implementation and I'm currently working on forking it to this project, but you can see if this works for you: https://github.com/SoylentGraham/Websocket-Sharp-UWP. I have tested it on the Hololens and it works no problem.

[arunavanag591]

Hi @jaguar243,

So did you just import the UWPWebSocketSharp.cs script and improt ROSSharpGetStarted.unitypackage or import ROSBridgeClient source code as a custom package and build it. My application is targetted for hololens for your information.

[rwarrier243]

@arunavanag591 you need to rebuild the websocket-sharp.dll with the uwp code interspersed with normal websockets using the target tags

#if UNITY_EDITOR
  <insert normal websocket code>
#else
  <insert uwp implementation>
#endif

at the places where the two implementions differ. I will update with a proof of concept soon but I'm currently working on just using a TCP client on the hololens connecting to a server on the PC sending data. Another issue with UWP is they don't allow loopback connections so your hololens and the server need to be connected to the same network and served with separate IP addresses. Hope that helps.

[Jasonthefirst]

@jaguar243 I followed the conversation and tried to implement this as well. I am not sure if I understand you correctly though. We need to rebuild the websocket-sharp.dll in the ros-sharp project or totally from weboscket-sharp scratch? Would this mean we use the /sta/websocket-sharp project, change all references depending if we use the normal implementation or the other one? Or do we only need to change the references in the ros-sharp project?
Thank you for your time and if I am totally wrong I can wailt for a Proof-of-concept in the future.

[rwarrier243]

@Jasonthefirst The websocket-sharp.dll needs to be rebuilt starting from /sta/websocket-sharp with the UWP implementation interspersed within that project using the target tags. The UWP implementation that I mentioned earlier maintains the same references so there are no changes required in the RosBridgeClient project. So the steps involved are:

1. Build websocket-sharp.dll with uwp implementation included.
2. Use the newly built websocket-sharp.dll in the RosBridgeClient VS solution and build the RosBridgeClient.dll
3. Include the newly built websocket-sharp.dll and RosBridgeClient.dll in the Unity project.

[arunavanag591]

@jaguar243 A quick question, while I trying to use the UWP version. The present websocket version should be working with UNITY EDITOR right? The UWP version comes into picture only when I am using it hololens?

I want it to work with the editor first.

Additionally is it possible to have a branch for UWP support containing the right websocket dll and rosbridgeclient dll

[rwarrier243]

@arunavanag591 Yes, on the editor the provided libraries work without any issues. The UWP build is for building and deploying to the hololens.

[Jasonthefirst]

@jaguar243 I was doing some other things so it took me a while to return to this again but I tried to work with what you told me the last few day.
I tried to change the websocket sharp but I think it didn't work because I wasn't able to build the rosbridgeclient.dll. Is it possible that you can provide me (and I guess a few others) with the changed dlls?) Would this even work?

[arunavanag591]

+1

[MartinBischoff]

Someone working with the Hololens/UWP and ROS# want to try this out?
https://github.com/SoylentGraham/Websocket-Sharp-UWP

[dwhit]

Hey so I got ros-sharp on the HoloLens! The trick I did was to add the SoylentGraham script above into unity, add the RosBridgeClient cs files to unity (removing the RosBridgeClient.dll), and wrap the appropriate code snipits in preprocessor directives. I'll cleanup the code and make a pull request.

[MartinBischoff]

These are very good news @dwhit !

Before including your modification in the master branch, we need to think about a way to include this alternative Websocket implementation properly.

ROS# is designed such that only all Unity-related code is included as .cs source files in Unity.
All generic code, that can be of use for other .NET applications beside Unity, is contained in the dlls.

• Instead of putting all RosBridgeClient scripts into Unity, would it be possible to provide a separate RosBridgeClientUWP.dll?
• Is it possible to switch between both Websocket implementations e.g. via a preprocessor directive in RosSocket?

In the meantime please feel free to provide any working implementation in your fork 👍

[dwhit]

Absolutely @MartinBischoff. My current implementation is too sloppy for mainbranch. I think the best way to do it would be to build two RosBridgeClient.dll files, and tag one to build in editor, and one to build to WSA. We already did this with Newtwonsoft.Json without any issues.

[MartinBischoff]

I am planning to write 3 implementations of a RosSharp.RosBridgeClient.Protocols.IProtocol interface:

• websocket-sharp
• Websocket-Sharp-UWP
• NET Websocket

Let us continue the discussion here.

[tarukosu]

Hi. This forked project supports for UWP and works in HoloLens.
https://github.com/tarukosu/ros-sharp/tree/support-for-uwp

[MartinBischoff]

Hi @tarukosu ! Thanks for this info and for sharing your code!
Likely there are several ROS# users who will give you thumbs up for this. We will take a look at your modificiations to see if we can take over anything in the major RosBridgeClient modification that is still pending #59 to be thoroughly tested, fixed and merged.

[tarukosu]

I know you are refactoring to support UWP. I'm looking forward to it.
I'm grad if my code is helpful!

[MartinBischoff]

Hi @tarukosu
and all UWP users!

With #59 done it is now an eay task to implement an IProtocol implementation for SoylentGraham's Websocket-sharp-UWP version that you are using in your fork.
I am not sure whether we should include this WebSocket client version in the master though.
Reasons:

• UWPWebsocket.cs looks not really cleaned up (i see a lot of outcommented code and todo comments)

• the existing WebSocketNetProtocol.cs might be aready sufficient. Can someone verify please?

In case someone wants to write the IProtocol implementation wrapping UWPWebsocket.cs anyway, it will look very similar to the wrapper for websocket-sharp.

[dwhit]

That's great news @MartinBischoff! I'll give it a spin now. A few months ago I wrote my own System.Net.Websocket based protocol, and although it worked in the editor, it didn't work on the HoloLens. Looking at the HoloLens documentation, it seems the only supported Websocket library is Windows.Networking.Sockets (despite claiming elsewhere that the HoloLens supports .NET Standard 2.0). So I'll try to write an IProtocol implementation of Websocket-sharp-UWP today! At the very least, it could live in another branch, or my fork.

[MartinBischoff]

Thank you @dwhit and @lee4138 for your UWP tests with these most recent ROS# changes.
I see you guys have big interest in getting ROS# operational for UWP. Though UWP-compatibility is of low importance for our team I am happy to support.

Apparently (and other than I have hoped it would be) the .NET ClientWebSocket class does not work with UWP (cf. Issue #59 ).

In order to get RosBridgeClient Websocket communication running for UWP you have the following options:

• Implement a WebSocketSharpUwpProtocol as we discussed in above two comments.
• Implement (or get a) a WebSocket Client based on Windows.NetWorking.Sockets and write a IProtocol wrapper for it.

@lee4138 informed us in issue #83 that he also has problems with other dependencies for UWP.
I know that we can relatively easily replace or get rid of all dependencies except Newtonsoft.Json in ROS#. @tarukosu maybe others have been successfully applying ROS# with UWP already. Can advise here?

[dwhit]

So I've written a WebSocket client based on Windows.Networking.Sockets that implements IProtocol. you can see it on my fork here. I'm currently testing now. The problem with it is that I don't see a way of compiling it into the RosBridgeClient.dll, since it requires a WindowsUWP compilation target, and the other WebSocket clients require .NET Framework 4.6 targets. I think the best solution will be to make two .dlls, and use specify which one include/exclude depending on the Unity Build Target (this can be done in the Unity editor).

As far as the other .dll dependencies, we can do roughly the same thing. I know Newtonsoft.Json has a UWP .dll, for instance.

[dwhit]

Okay, so I've tested my code, and my WebSocketUWPProtocol.cs file works. I have the working branch here.

In this branch, I removed the RosBridgeClient.dll and UrdfImporter.dll and added all source code to the unity project, to streamline my development. If this branch were ever to be merged into master, the next step would be moving the Library code back out of Unity, and setting up a UWP build target for RosBridgeClient.dll.

Even if we did that though, there would still need to be preprocessor directives peppered all over the RosBridgeClient object. I can't think of a way around that.

[linhdoan8]

Hi all,
thanks very much @MartinBischoff for your support and @tarukosu and @dwhit for sharing your code !
I tested the version of @tarukosu and it works perfectly fine with a UWP build on Hololens! @dwhit please give it a try! Somehow the library Newtonsoft.Json is still in there and does not give any errors.

[MartinBischoff]

Great job @dwhit !
I'm happy to see this IProtocol implementation you wrote. This, and possible future IProtocol implementations, is the reason we designed this interface.
The UWP platform support was requested by several ROS# users. It enables a variety of new applications.

We now proceed as follows:

• The working UWP implementation stays on dwhit's fork. Adding it on the master would involve additional engineering and testing effort and complicate the project (many preprocessor directives, platform-specific assemblies, etc.)

• We add WebSocketUWPProtocol.cs to the master and provide a Wiki page which shall enable users to convert this .NET Platform version into a UWP version themselves.

[tarukosu]

Hi all!
I wrote a blog about the forked version of ROS# (I'm sorry, but this is written in Japanese).
https://tarukosu.hatenablog.com/entry/RosSharpUWP

[MartinBischoff]

@tarukosu nice!
Ìf you translate this page into English, we can refer to it from within the ROS# Wiki.

[dwhit]

Hey all, my fork is done! You can view it here: https://github.com/dwhit/ros-sharp

I edited the main Readme with instructions for installation, and an overview of the architecture for others who would want to edit or extend it.

[linhdoan8]

Hi @dwhit,
wow looks really good! I will try this out. Thanks for sharing!

[MartinBischoff]

Beautiful @dwhit ! I just updated the link on README.md to the master branch of your fork. From my perspective this issue is done.

@dwhit if you agree, please close it.

[dwhit]

I agree it should be closed, but I didn't actually open it, @arunavanag591 did.

[lotfielhafi]

@dwhit thank you so much for your UWP implementation of ROS#. I would like to ask you one question regarding your fork. According to the readme, Unity 2018.x is required but it is not clear if the MixedRealityToolkit-Unity can support higher Unity versions than 2017.x. Could you share with us what configuration of Unity and MixedRealityToolkit-Unity you have tested and confirmed working? Maybe this should be included into the readme of your fork? Thank you in advance for your help.

[dwhit]

Hi @lotfielhafi!, I can run the MixedRealityToolkit without issues on 2018.2. I just followed the install instructions from the Getting Started page. I'll add this info to the Readme!

[roalchaq]

Hi all,

I just went through the process of integrating ROS# with Unity 2018.3 for the HoloLens and wanted to share what I did as it is a bit different from what is described by @dwhit in his fork:

1. The MRTK Latest Release (2017.4.3.0) is not supported in Unity 2018.3, and more recent versions (such as vNext and v2) no longer includes the Newtonsoft.Json.dll needed. In addition, ".NET Scripting Backend for UWP has been deprecated and will be removed in a future Unity release. We recommend switching to IL2CPP scripting backend.". To solve this add the Newtonsoft.dll for AOT from this unity package.

2. As described in issue #184, communication.cs needs to be changed: instead of using public override string op { get { return "advertise"; } } use this.op ="advertise" etc...

3. Build rossharp.sln using Windows SDK 18362+ (as recommended here) to include these changes and update both RosBridgeClient.dll and RosBridgeClientUWP.dll inside the RosSharp > Plugins folder.

[acnacor]

Hey @roalchaq, my team and I are currently stuck on a project and we're wondering if we can ask you a few questions. I can't seem to find any contact details on your profile, please feel free to message me.