Using rs2::gl::align produces half images

[xsol]

| Required Info
|---------------------------------|------------------------------------------- |
| Camera Model | { D455 } |
| Firmware Version | 05.14.00.00|
| Operating System & Version | Ubuntu 20.04LTS |
| Kernel Version (Linux Only) | 5.15.0 | |
| Platform | PC, NVIDIA Jetson AGX XAVIER |
| SDK Version | 2.53.1 |
| Language | C++ |
| Segment | Robot |

Issue Description

Hello, i'm trying to get the rs-align example to work with glsl gpu processing to fix the issue of low framerate/high delays when using the regular rs2::align on the Jetson AGX Xavier as discussed here: IntelRealSense/realsense-ros#2396. I modified the example code (see below), and i'm using a D455.

The Problem is the following: not only is the image after alignment not aligned, the left half is completely blank- and the right half should be shifted to the left. These images taken from the modified rs-align will make it clear:

Aligning to color, displaying the color image:

Aligning to color, displaying the depth image:

As can bee seen, the depth image only shows the door that is on the left side of the color image. But It is shifted too far right and the flower pot is missing from the depth image completely.

modified rs-align example (modifications are marked by comment, only changed main function and additional include):

#include <librealsense2-gl/rs_processing_gl.hpp>             // MODIFICATION
int main(int argc, char * argv[]) try
{
    std::string serial;
    if (!device_with_streams({ RS2_STREAM_COLOR,RS2_STREAM_DEPTH }, serial))
        return EXIT_SUCCESS;

    // Create and initialize GUI related objects
    window app(1280, 720, "RealSense Align Example"); 
    ImGui_ImplGlfw_Init(app, false);      
    rs2::gl::colorizer c;                                                           // MODIFICATION
    texture depth_image, color_image;   
    rs2::gl::init_processing(app, true);                                   // MODIFICATION
    // Initialize rendering module:
    rs2::gl::init_rendering();                                                   // MODIFICATION

    // Create a pipeline to easily configure and start the camera
    rs2::pipeline pipe;
    rs2::config cfg;
    if (!serial.empty())
        cfg.enable_device(serial);
    cfg.enable_stream(RS2_STREAM_DEPTH);
    cfg.enable_stream(RS2_STREAM_COLOR);
    pipe.start(cfg);

    // Define two align objects. One will be used to align
    // to depth viewport and the other to color.
    // Creating align object is an expensive operation
    // that should not be performed in the main loop
    rs2::gl::align align_to_depth(RS2_STREAM_DEPTH);          // MODIFICATION
    rs2::gl::align align_to_color(RS2_STREAM_COLOR);           // MODIFICATION

    float       alpha = 0.5f;               // Transparancy coefficient
    direction   dir = direction::to_depth;  // Alignment direction

    while (app) // Application still alive?
    {
        // Using the align object, we block the application until a frameset is available
        rs2::frameset frameset = pipe.wait_for_frames();

        if (dir == direction::to_depth)
        {
            // Align all frames to depth viewport
            frameset = align_to_depth.process(frameset);
        }
        else
        {
            // Align all frames to color viewport
            frameset = align_to_color.process(frameset);
        }

        // With the aligned frameset we proceed as usual
        auto depth = frameset.get_depth_frame();
        auto color = frameset.get_color_frame();
        auto colorized_depth = c.colorize(depth);

        glEnable(GL_BLEND);
        // Use the Alpha channel for blending
        glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);

        if (dir == direction::to_depth)
        {
            // When aligning to depth, first render depth image
            // and then overlay color on top with transparancy
            depth_image.render(colorized_depth, { 0, 0, app.width(), app.height() });
            color_image.render(color, { 0, 0, app.width(), app.height() }, alpha);
        }
        else
        {
            // When aligning to color, first render color image
            // and then overlay depth image on top
            color_image.render(color, { 0, 0, app.width(), app.height() });
            depth_image.render(colorized_depth, { 0, 0, app.width(), app.height() }, 1 - alpha);
        }

        glColor4f(1.f, 1.f, 1.f, 1.f);
        glDisable(GL_BLEND);

        // Render the UI:
        ImGui_ImplGlfw_NewFrame(1);
        render_slider({ 15.f, app.height() - 60, app.width() - 30, app.height() }, &alpha, &dir);
        ImGui::Render();
    }

    return EXIT_SUCCESS;
}
catch (const rs2::error & e)
{
    std::cerr << "RealSense error calling " << e.get_failed_function() << "(" << e.get_failed_args() << "):\n    " << e.what() << std::endl;
    return EXIT_FAILURE;
}
catch (const std::exception & e)
{
    std::cerr << e.what() << std::endl;
    return EXIT_FAILURE;
}

Since there is little documentation i could find on rs2::gl, i'm not sure if i missed a critical step for the alignment to work. It seems like during alignment, the depth image/ the pointcloud calculated from depth are shifted too far right- thus the blank left side, maybe something with applying the extrinsic matrix? Or maybe some issue while writing into the aligned image? But that's just speculation, any help is appreciated.

[MartyG-RealSense]

Hi @xsol A RealSense user attempted to implement alignment with rs2::gl a few years ago at the link below and the output was a black screen.

bmegli/realsense-network-hardware-video-encoder#24

[xsol]

Are you aware of a working example of rs2::gl::align?
If not, i'll try to create my own align function which uses rs2::gl::pointcloud (that's working fine) and does the remaining calculation on cpu.

[MartyG-RealSense]

There are not any other references for rs2::gl::align other than the one that I linked to, unfortunately.

[MartyG-RealSense]

Hi @xsol Bearing in mind the comment above, do you require further assistance with this case please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.

[Arun-Prasad-V]

Hi @xsol, this half image issue from rs2::gl::align has been fixed now and available in development branch.
Please refer the PR #12556.

Thanks.