Obtaining RGB values fora colorised PCL point cloud.

[conorjmcq]

|---------------------------------|------------------------------------------- |
| Camera Model | {SR300} |
| Operating System & Version | Win (10) |
| Platform | PC|
| SDK Version | { 2.10.3 } |

Issue Description

Hi there,

I'm using the SR300 for a college assignment and am stuck on a bit of an issue.

I am using PCL and have generated the required x, y, z coords for the pointcloud. However I would like to manipulate the data that I have captured depending on the individual RGB values at the desired point. I understand that get_texture_coordinates will return the U, V coords but is there a way to only extract the RGB values, or even obtain them somehow ?

Thank you very much for you time.

[dorodnic]

Hi @conorjmcq
We don't have such an example out of the box, but the process should be very similar to colorized PLY export
More specifically please refer to archive.cpp:57
P.S. If you solve the problem, please consider contributing to the project to help future developers.

[conorjmcq]

Hi @dorodnic ,

Thank you for the quick reply. I haven't managed to come to a solution yet unfortunately. You're link definitely helped me to understand the method, but I'm still miles off.

#1143

This link talks about the same issue but I don't think there's much mention of how they achieved it.

If there is anything else you could help me with then I'd greatly appreciate it. If not then thank you very much for your time.

[Yonatan-Sade]

Hi. I am still struggling with it...
Can anyone help me with this? It would be very appreciated...
I want to convert data from the camera to XYZRGB point cloud of PCL.

I am starting with:

auto frames = pipe.wait_for_frames();
auto depth = frames.get_depth_frame();
points = pc.calculate(depth);
auto color = frames.get_color_frame();
pc.map_to(color);
auto vertices = points.get_vertices();      
auto tex_coords = points.get_texture_coordinates();

Then, using the vertices I can assign the XYZ values.
But How can I get the RGB?
I know that can be done by the color and tex_coords, but how excactly?
thank all for any support

Yonatan

[Yonatan-Sade]

I think I got it!
Try this code snippet. worked for me (not all the points got valid color - any idea?):

#include <librealsense2/rs.hpp> // Include RealSense Cross Platform API
#include "../../../examples/example.hpp" // Include short list of convenience functions for rendering
#include <pcl/point_types.h>
#include <pcl/filters/passthrough.h>
#include <pcl/visualization/pcl_visualizer.h>

std::tuple<uint8_t, uint8_t, uint8_t> get_texcolor(rs2::video_frame texture, rs2::texture_coordinate texcoords)
{
	const int w = texture.get_width(), h = texture.get_height();
	int x = std::min(std::max(int(texcoords.u*w + .5f), 0), w - 1);
	int y = std::min(std::max(int(texcoords.v*h + .5f), 0), h - 1);
	int idx = x * texture.get_bytes_per_pixel() + y * texture.get_stride_in_bytes();
	const auto texture_data = reinterpret_cast<const uint8_t*>(texture.get_data());
	return std::tuple<uint8_t, uint8_t, uint8_t>(
		texture_data[idx], texture_data[idx + 1], texture_data[idx + 2]);
}

pcl::PointCloud<pcl::PointXYZRGB>::Ptr points_to_pcl(const rs2::points& points, const rs2::video_frame& color)
{
	pcl::PointCloud<pcl::PointXYZRGB>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZRGB>);

	auto sp = points.get_profile().as<rs2::video_stream_profile>();
	cloud->width = sp.width();
	cloud->height = sp.height();
	cloud->is_dense = false;
	cloud->points.resize(points.size());

	auto tex_coords = points.get_texture_coordinates();
	auto vertices = points.get_vertices();

	for (int i = 0; i < points.size(); ++i)
	{
		cloud->points[i].x = vertices[i].x;
		cloud->points[i].y = vertices[i].y;
		cloud->points[i].z = vertices[i].z;

		std::tuple<uint8_t, uint8_t, uint8_t> current_color;
		current_color = get_texcolor(color, tex_coords[i]);

		cloud->points[i].r = std::get<0>(current_color);
		cloud->points[i].g = std::get<1>(current_color);
		cloud->points[i].b = std::get<2>(current_color);
	}

	return cloud;
}

int main(int argc, char * argv[])
{
	boost::shared_ptr<pcl::visualization::PCLVisualizer> viewer(new pcl::visualization::PCLVisualizer("3D Viewer"));

	// Declare pointcloud object, for calculating pointclouds and texture mappings
	rs2::pointcloud pc;
	// We want the points object to be persistent so we can display the last cloud when a frame drops
	rs2::points points;

	// Declare RealSense pipeline, encapsulating the actual device and sensors
	rs2::pipeline pipe;
	// Start streaming with default recommended configuration
	pipe.start();

	while (!viewer->wasStopped())
	{
		// Wait for the next set of frames from the camera
		auto frames = pipe.wait_for_frames();
		auto depth = frames.get_depth_frame();
		auto color = frames.get_color_frame();
		
		// Generate the pointcloud and texture mappings
		points = pc.calculate(depth);
		pc.map_to(color);

		// Convert to PCL: 
		auto pcl_points = points_to_pcl(points, color);

		viewer->removeAllPointClouds();
		viewer->addPointCloud<pcl::PointXYZRGB>(pcl_points);

		viewer->spinOnce(100);
		boost::this_thread::sleep(boost::posix_time::microseconds(1));
	}

	return EXIT_SUCCESS;
}

[conorjmcq]

Hi there, I gave up that pursuit and decided that I was going to color the points based on depth. It turned out how I wanted it, but thank you very much for sending this code chunk. I don't know how helpful I may be as I still have very little experience. I'm currently trying to implement SLAM with the Realsense. I'll try your code snippet once I get into College. Thanks again! Conor.

…

On 8 May 2018 at 20:19, Yonatan Sade ***@***.***> wrote: I think I got it! Try this code snippet. worked for me (not all the points got valid color - any idea?): #include <librealsense2/rs.hpp> // Include RealSense Cross Platform API #include "../../../examples/example.hpp" // Include short list of convenience functions for rendering #include <pcl/point_types.h> #include <pcl/filters/passthrough.h> #include <pcl/visualization/pcl_visualizer.h> std::tuple<uint8_t, uint8_t, uint8_t> get_texcolor(rs2::video_frame texture, rs2::texture_coordinate texcoords) { const int w = texture.get_width(), h = texture.get_height(); int x = std::min(std::max(int(texcoords.u*w + .5f), 0), w - 1); int y = std::min(std::max(int(texcoords.v*h + .5f), 0), h - 1); int idx = x * texture.get_bytes_per_pixel() + y * texture.get_stride_in_bytes(); const auto texture_data = reinterpret_cast<const uint8_t*>(texture.get_data()); return std::tuple<uint8_t, uint8_t, uint8_t>( texture_data[idx], texture_data[idx + 1], texture_data[idx + 2]); } pcl::PointCloud<pcl::PointXYZRGB>::Ptr points_to_pcl(const rs2::points& points, const rs2::video_frame& color) { pcl::PointCloud<pcl::PointXYZRGB>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZRGB>); auto sp = points.get_profile().as<rs2::video_stream_profile>(); cloud->width = sp.width(); cloud->height = sp.height(); cloud->is_dense = false; cloud->points.resize(points.size()); auto tex_coords = points.get_texture_coordinates(); auto vertices = points.get_vertices(); for (int i = 0; i < points.size(); ++i) { cloud->points[i].x = vertices[i].x; cloud->points[i].y = vertices[i].y; cloud->points[i].z = vertices[i].z; std::tuple<uint8_t, uint8_t, uint8_t> current_color; current_color = get_texcolor(color, tex_coords[i]); cloud->points[i].r = std::get<0>(current_color); cloud->points[i].g = std::get<1>(current_color); cloud->points[i].b = std::get<2>(current_color); } return cloud; } int main(int argc, char * argv[]) { boost::shared_ptr<pcl::visualization::PCLVisualizer> viewer(new pcl::visualization::PCLVisualizer("3D Viewer")); // Declare pointcloud object, for calculating pointclouds and texture mappings rs2::pointcloud pc; // We want the points object to be persistent so we can display the last cloud when a frame drops rs2::points points; // Declare RealSense pipeline, encapsulating the actual device and sensors rs2::pipeline pipe; // Start streaming with default recommended configuration pipe.start(); while (!viewer->wasStopped()) { // Wait for the next set of frames from the camera auto frames = pipe.wait_for_frames(); auto depth = frames.get_depth_frame(); auto color = frames.get_color_frame(); // Generate the pointcloud and texture mappings points = pc.calculate(depth); pc.map_to(color); // Convert to PCL: auto pcl_points = points_to_pcl(points, color); viewer->removeAllPointClouds(); viewer->addPointCloud<pcl::PointXYZRGB>(pcl_points); viewer->spinOnce(100); boost::this_thread::sleep(boost::posix_time::microseconds(1)); } return EXIT_SUCCESS; } — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1601 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/Ae1WhCDEHGwk8_4cg1bAYHA3yYW_83tSks5twe--gaJpZM4TiUq5> .

[GKachaner]

Hello !
I had the same issue and i solved it using this function :

typedef pcl::PointXYZRGB PointT;
pcl::PointCloud<PointT>::Ptr points_to_pcl(const rs2::points& points)
{
	pcl_ptr cloud(new pcl::PointCloud<PointT>);


	auto sp = points.get_profile().as<rs2::video_stream_profile>();
	cloud->width = sp.width();
	cloud->height = sp.height();
	cloud->is_dense = false;
	cloud->points.resize(points.size());
	auto ptr = points.get_vertices();
	for (auto& p : cloud->points)
	{
		p.x = ptr->x;
		p.y = ptr->y;
		p.z = ptr->z;
		ptr++;
	}

	return cloud;
}

The trick is in the type of points you are using with PCL.
If you stick to PointXYZ you won't store the color information.
Using PointXYZRGB should do it.
Hope it works for you.

Grégoire

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
@conorjmcq
Can we close the issue if you have found solution?

[conorjmcq]

Yes we can, thank you very much everyone !