Error in Reconstruction System using Intel RealSense Depth Camera D435i

[ridanlukita]

Describe the bug
I had some issues when trying the reconstruction system using Intel RealSense D435i camera.

After I try both of the following Input arguments below (one by one):
python realsense_recorder.py --record_imgs
python realsense_recorder.py --record_rosbag

the color and depth folder can be produced by python realsense_recorder.py --record_imgs
and also realsense.bag can be produced by python realsense_recorder.py --record_rosbag

But when I try to:

• Make zlib/jpeg/png/... auto link to native libraries on Ubuntu/OSX #1. Run reconstruction system from RGBD dataset (color and depth folder) like in the Quick start
  python run_system.py config_file --make --register --refine --integrate
  where my realsense.json configuration is:

{
    "name": "Realsense bag file",
    "path_dataset": "dataset/realsense",
    "path_intrinsic": "",
    "max_depth": 3.0,
    "voxel_size": 0.05,
    "max_depth_diff": 0.07,
    "preference_loop_closure_odometry": 0.1,
    "preference_loop_closure_registration": 5.0,
    "tsdf_cubic_size": 3.0,
    "icp_method": "color",
    "global_registration": "ransac",
    "python_multi_threading": true
}

It runs pretty well and producing like the Results with my own dataset
But it was blur and had distortion even when I do record with slow, steady, and longer time.
And sometimes it creates scattered point cloud (shown in the screenshots with blue and red mark as stuff that I have scanned).

• Add Text rendering support to Visualizer #2. Run reconstruction system from the realsense.bag with realsense.json configuration below:

{
    "name": "Realsense bag file",
    "path_dataset": "dataset/realsense/realsense.bag",
    "path_intrinsic": "",
    "max_depth": 3.0,
    "voxel_size": 0.05,
    "max_depth_diff": 0.07,
    "preference_loop_closure_odometry": 0.1,
    "preference_loop_closure_registration": 5.0,
    "tsdf_cubic_size": 3.0,
    "icp_method": "color",
    "global_registration": "ransac",
    "python_multi_threading": true
}

the script python run_system.py config/realsense.json --make --register --refine --integrate was resulting error like this:

Extracting frames from RGBD video file
../utility\file.py:126: UserWarning: Skipping frame extraction for dataset/realsense/realsense.bag since files are present.
  warn(f"Skipping frame extraction for {rgbd_video_file} since files are"
====================================
Configuration
====================================
                                    name : Realsense bag file
                            path_dataset : dataset/realsense\realsense
                          path_intrinsic : dataset/realsense\realsense\intrinsic.json
                               max_depth : 3.0
                              voxel_size : 0.05
                          max_depth_diff : 0.07
        preference_loop_closure_odometry : 0.1
    preference_loop_closure_registration : 5.0
                         tsdf_cubic_size : 3.0
                              icp_method : color
                     global_registration : ransac
                  python_multi_threading : True
                          depth_map_type : redwood
                   n_frames_per_fragment : 100
                 n_keyframes_per_n_frame : 5
                               min_depth : 0.3
                             depth_scale : 999.9999389648438
                          max_iterations : 5
                               sdf_trunc : 0.04
                             block_count : 40000
                      distance_threshold : 0.07
                       fitness_threshold : 0.3
                      regularizer_weight : 1
                                  method : slac
                                  device : cpu:0
                          save_output_as : pointcloud
                             folder_slac : slac/
       template_optimized_posegraph_slac : optimized_posegraph_slac.json
                         folder_fragment : fragments/
                          subfolder_slac : slac/0.050/
             template_fragment_posegraph : fragments/fragment_%03d.json
   template_fragment_posegraph_optimized : fragments/fragment_optimized_%03d.json
            template_fragment_pointcloud : fragments/fragment_%03d.ply
                            folder_scene : scene/
               template_global_posegraph : scene/global_registration.json
     template_global_posegraph_optimized : scene/global_registration_optimized.json
              template_refined_posegraph : scene/refined_registration.json
    template_refined_posegraph_optimized : scene/refined_registration_optimized.json
                    template_global_mesh : scene/integrated.ply
                    template_global_traj : scene/trajectory.log
                              debug_mode : False
OpenCV is detected. Using ORB + 5pt algorithm
making fragments from RGBD sequence.
Traceback (most recent call last):
  File "run_system.py", line 91, in <module>
    make_fragments.run(config)
  File "D:\E-Learning\Visual Studio 2019\source\repos\Open3D\examples\python\reconstruction_system\make_fragments.py", line 
185, in run
    Parallel(n_jobs=MAX_THREAD)(delayed(process_single_fragment)(
  File "C:\ProgramData\Anaconda3\lib\site-packages\joblib\parallel.py", line 966, in __call__
    n_jobs = self._initialize_backend()
  File "C:\ProgramData\Anaconda3\lib\site-packages\joblib\parallel.py", line 733, in _initialize_backend
    n_jobs = self._backend.configure(n_jobs=self.n_jobs, parallel=self,
  File "C:\ProgramData\Anaconda3\lib\site-packages\joblib\_parallel_backends.py", line 489, in configure
    n_jobs = self.effective_n_jobs(n_jobs)
  File "C:\ProgramData\Anaconda3\lib\site-packages\joblib\_parallel_backends.py", line 504, in effective_n_jobs
    raise ValueError('n_jobs == 0 in Parallel has no meaning')
ValueError: n_jobs == 0 in Parallel has no meaning

To Reproduce
Steps to reproduce the behavior:

• for issue Make zlib/jpeg/png/... auto link to native libraries on Ubuntu/OSX #1:

1. cd 'the path to reconstruction_system/sensors'
2. python python realsense_recorder.py --record_imgs
3. capturing, after that, close with press esc (I do scanning until ±6000 color + depth image captured)
4. realsense folder created

realsense\
    color\rgb_files.jpg
    depth\depth_files.png
    camera_intrinsic.json

where camera_instrinsic.json by default is:

{
    "width": 640,
    "height": 480,
    "intrinsic_matrix": [
        613.3899536132812,
        0,
        0,
        0,
        613.7398071289062,
        0,
        328.3892822265625,
        245.40103149414062,
        1
    ]
}

5. cd ..
6. python python run_system.py config/realsense.json --make --register --refine --integrate
   with realsense.json configuration:

{
    "name": "Realsense bag file",
    "path_dataset": "dataset/realsense",
    "path_intrinsic": "",
    "max_depth": 3.0,
    "voxel_size": 0.05,
    "max_depth_diff": 0.07,
    "preference_loop_closure_odometry": 0.1,
    "preference_loop_closure_registration": 5.0,
    "tsdf_cubic_size": 3.0,
    "icp_method": "color",
    "global_registration": "ransac",
    "python_multi_threading": true
}        

7. fragments and scene folder created
8. I create visualize_point_cloud.ipynb which inside were extracted from Visualize point cloud and change the read_point_cloud path to ../reconstruction_system/dataset/realsense/scene/integrated.ply
9. The result was in the scattered point cloud screenshots with blue and red marks

• for issue Add Text rendering support to Visualizer #2:

1. cd 'the path to reconstruction_system/dataset'
2. mkdir realsense
3. cd ../sensors
4. python python realsense_recorder.py --record_rosbag
5. capturing, after that, close with press esc
6. realsense.bag created in reconstruction_system/dataset/realsense
7. configure path_dataset in realsense.json from "dataset/realsense" to "dataset/realsense/realsense.bag"
8. cd ..
9. python python run_system.py config/realsense.json --make --register --refine --integrate
10. resulting an error as mentioned above (ValueError: n_jobs == 0 in Parallel has no meaning), with new realsense directory like below:

realsense\
    realsense\
        color\
        depth\
        fragments\
        intrinsic.json
    realsense.bag

where intrinsic.json by default is:

{
	"color_format" : "BGR8",
	"depth_format" : "Z16",
	"depth_scale" : 999.99993896484375,
	"device_name" : "Intel RealSense D435I",
	"fps" : 30.0,
	"height" : 480,
	"intrinsic_matrix" : 
	[
		613.38995361328125,
		0.0,
		0.0,
		0.0,
		613.73980712890625,
		0.0,
		328.3892822265625,
		245.40103149414063,
		1.0
	],
	"serial_number" : "918512071170",
	"stream_length_usec" : 4946741,
	"width" : 640
}

Expected behavior

• for issue Make zlib/jpeg/png/... auto link to native libraries on Ubuntu/OSX #1: pointcloud constructed without scattered and the result isn't blur or distorted
• for issue Add Text rendering support to Visualizer #2: does it should be like the documentation Running the scence reconstruction pipeline (with my own .bag file)? if yes, then how to do that using my own .bag file?

Screenshots

Environment:

• Operating system: Windows 10 64-bit
• Python version: Python 3.8
• Open3D version: 0.12.0
• Is this remote workstation?: no
• How did you install Open3D?: pip

Additional context

• Do you have suggestion to repair the scattered pointcloud and error (from .bag file) above?
• Any configuration needed that should I set for the camera?

[lhamza98]

Hi @ridanlukita . Did you end up solving the second problem ?
I seem to be having the same error as well.

[ridanlukita]

@lhamza98 for the .bag file, I still cannot read the file, therefore, I use the color and depth file instead.

[darldsla]

for issue " ValueError: n_jobs == 0 in Parallel has no meaning"

• lower fps from 30, make sure depth and rgb have same fps in the recorded .bag
  ref: http://www.open3d.org/docs/release/python_api/open3d.t.io.RSBagReader.html

[akash02ita]

@darldsla for

ValueError: n_jobs == 0 in Parallel has no meaning

I have the exact issue as well @ridanlukita .
I am using realsense d435i with python realsense_recorder --record_imgs and the appropriate realnsense.json config file just compiler the reconstruction pretty well.

But when using the realsense.json config file for python realsense_recorder --record_rosbag, indeed there is a problem at
extract_rgbd_frames() in initialize_config.py. However, based on @darldsla reply i wonder if now issue is in initialize_config.py or realsense_recorder.py.

If realsense_recorder.py is the issue, then where do i need to make code changes?

[akash02ita]

@darldsla for

ValueError: n_jobs == 0 in Parallel has no meaning

I have the exact issue as well @ridanlukita . I am using realsense d435i with python realsense_recorder --record_imgs and the appropriate realnsense.json config file just compiler the reconstruction pretty well.

But when using the realsense.json config file for python realsense_recorder --record_rosbag, indeed there is a problem at extract_rgbd_frames() in initialize_config.py. However, based on @darldsla reply i wonder if now issue is in initialize_config.py or realsense_recorder.py.

If realsense_recorder.py is the issue, then where do i need to make code changes?

Also, i just tested it out

I still wonder if realsense_recorder.py has any issues.