Unity Rgb-d Simulated Camera to ROS

[panagelak]

Hello i really like this project!

I believe the Community miss an easy example/functionality of simulating an rgbd camera in Unity and transferring the topics to Ros efficiently.

I have created a starter project for this purpose here : https://github.com/panagelak/rgbd_unity_camera

I believe the best way to do that is to send into ros 4 topics namely

1. a compress rgb image and camera info topic (which i have successfully done)
2. a compress depth image and depth camera info topic ( which i need help with!!)

Then we can decompress the rgb and depth images into ros and combine them with the camera info topics to construct colored or not pointclouds, through the image transport packages.

For convenience i have included this packages from the ros image_transport packages, which i have modified slightly in order to include them as libraries.

I tried to do it by myself but unfortunately i am not very familiar with Unity and shaders : (

I also included some code from here : https://www.immersivelimit.com/tutorials/unity-depth-camera-simulation
But the depth image i managed to get was a gray image of rgb format instead of the 32FC1 that ros expects in depth images.

So i think the difficult part will be to do the following two things

1. Create an example of a unity depth camera that uses a shader with 32FC1 Format.
2. Create an Encode (compress) algorithm that compresses this format, like the data = texture2D.EncodeToJPG(qualityLevel)

If somebody can help me with that, that would be great!

You can make me a pull request on my repo for now (ros-noetic, Unity 2020.3.25), in order to not have to meet high standards but just a working code
and later i think it would be a great addition to have as an example (along with ros actions P)

Thank you,
I am awaiting your input

[github-actions]

This issue has been marked stale because it has been open for 14 days with no activity. Please remove the stale label or comment on this issue, or the issue will be automatically closed in the next 14 days.

[vidurvij-Unity]

Thank you posting this. I will look into this issue.

[panagelak]

@vidurvij-Unity

Thank you for taking the time

[ZhuoyunZhong]

Do you have the problem of frame drop when you publish images to ROS and show them in Rviz? My framerate can drop from ~200Hz to 40Hz when publishing images.

[florianfhws]

Dear @vidurvij-Unity i am working on this problem as well, for me the problem is that the publishing and the rendering of the cameras is done in the same function and therefor my depth camera to wait till the color camera is finished. I know that rendering is bound to one thread so stacking an infinite amount of cameras is not possible.
However is it possible to put the publishing of the images in a separate thread that is called from the function that renders the camera image?
So for clarification
I have two cameras on each of the cameras is a script with a InvokeRepeating function in these function the camera image gets rendered and then put into a ros message and then transmitted. Lets call the functions cam1func and cam2func.
So the problem is cam2func has to wait till cam1func is fully procesed to start even so the rendering part of cam1func which blocks the mainthread is done cam2func will not start.

[florianfhws]

Right now another problem is that packing the image data into a message is taking up a lot of time (20-30 ms) by EncodeToJPG() as the data should be compressed to avoid huge data to be transferred. can this process also be packed into a thread?

[panagelak]

@florianfhws

you might be interested in the implementation at https://github.com/fsstudio-team/ZeroSimROSUnity
(there they don't compress it and i had to run the rosbridge from the provided docker after i fixed some minor build errors)

also i raised an issue with a problem on the distortion of the pointcloud (due to what i believe incorrect depth data representation and possibly cam info)

fsstudio-team/ZeroSimROSUnity#20

also the format of the depth image should be 32fc1 and i'm not sure if you can use EncodeToJPG()

[florianfhws]

@panagelak the project is very interesting especially the use of AsyncGPUReadbackRequest.
However as the request only enables access to the raw data you would first need to copy them into a Texture2D and then call the encodetojpg function and again encodetojpg would cause the mainthread to wait slow down (15 ms), so packing raw data into your messages is the only fast option.
A solution with packing the raw data into a opencv matrix and then converting it also slowed down the callback of AsyncGPUReadbackRequest too much (3-4 ms).
Both values when wanting to render and therefor calling AsyncGPUReadbackRequest every 0.05s