failed to use my own data

[Ian-YJX]

When I used my own rosbag (including rgb images, depth images, camera_info and tf_statics) and edited the configuation file, I failed to see the reconstruction results. What's the problem here?

[maxwildersmith]

Hmm, try saving the data out to files and running in offline mode to manually check the RGB images are valid and if its an issue with the method

[Ian-YJX]

When I used offline mode, the "ns-train" command had a result like this:
(nerfstudio) root@lab404:/mnt/home/lab404/YJX/ros1_ws# ns-train depth-nerfacto nerfstudio-data --orientation-method none --data data/2024-11-06_11-23-07 ...... [11:28:31] Saving config to: outputs/unnamed/depth-nerfacto/2024-11-06_112831/config.yml experiment_config.py:136 Saving checkpoints to: outputs/unnamed/depth-nerfacto/2024-11-06_112831/nerfstudio_models trainer.py:137 Auto image downscale factor of 1 nerfstudio_dataparser.py:484 Traceback (most recent call last): File "/root/miniconda/envs/nerfstudio/bin/ns-train", line 8, in <module> sys.exit(entrypoint()) File "/root/miniconda/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 262, in entrypoint main( File "/root/miniconda/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 247, in main launch( File "/root/miniconda/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 189, in launch main_func(local_rank=0, world_size=world_size, config=config) File "/root/miniconda/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 99, in train_loop trainer.setup() File "/root/miniconda/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/engine/trainer.py", line 153, in setup self.pipeline = self.config.pipeline.setup( File "/root/miniconda/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/configs/base_config.py", line 54, in setup return self._target(self, **kwargs) File "/root/miniconda/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/pipelines/base_pipeline.py", line 255, in __init__ self.datamanager: DataManager = config.datamanager.setup( File "/root/miniconda/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/configs/base_config.py", line 54, in setup return self._target(self, **kwargs) File "/root/miniconda/envs/nerfstudio/lib/python3.8/typing.py", line 729, in __call__ result = self.__origin__(*args, **kwargs) File "/root/miniconda/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/data/datamanagers/base_datamanager.py", line 404, in __init__ self.train_dataparser_outputs: DataparserOutputs = self.dataparser.get_dataparser_outputs(split="train") File "/root/miniconda/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/data/dataparsers/base_dataparser.py", line 165, in get_dataparser_outputs dataparser_outputs = self._generate_dataparser_outputs(split, **kwargs) File "/root/miniconda/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/data/dataparsers/nerfstudio_dataparser.py", line 246, in _generate_dataparser_outputs scale_factor /= float(torch.max(torch.abs(poses[:, :3, 3]))) ZeroDivisionError: float division by zero

When I checked the transforms.json, I found the problem should be caused by the transform matrix, it goes like this:
"camera_model": "OPENCV", "frames": [ { "file_path": "./rgb/arm_00035.png", "depth_file_path": "./depth/arm_00037.png", "transform_matrix": [ [ 6.123234262925839e-17, 6.123234262925839e-17, -1.0, 0.0 ], [ -1.0, 3.7493993930529855e-33, -6.123234262925839e-17, 0.0 ], [ 0.0, 1.0, 6.123234262925839e-17, 0.0 ], [ 0.0, 0.0, 0.0, 1.0 ] ] },
The forth column of every matrix is "0 0 0 1". What can I do to stop this?

[maxwildersmith]

4th column should be 0 0 0 1 as its homogenous coordinates as shown in the docs. However the rest of the poses seem quite small...

You can disable the pose scaling with the argument --auto-scale-poses False. I would recomend checking the rgb and depth images to make sure they look correct.