load_save_viewpoint (and other custom visualization scripts) not producing expected behavior

[natesimon]

Checklist

• I have searched for similar issues.
• For Python issues, I have tested with the latest development wheel.
• I have checked the release documentation and the latest documentation (for master branch).

My Question

Hello! ctr.convert_from_pinhole_camera_parameters(param) does not update the vis object for me. The simplest example of this issue is demonstrated by load_save_viewpoint.py, where the viewpoint is properly loaded but not reflected in the updated visualization.

I added the following print statements:

def load_view_point(pcd, filename):
    vis = o3d.visualization.Visualizer()
    vis.create_window()
    ctr = vis.get_view_control()
    param = o3d.io.read_pinhole_camera_parameters(filename)
    vis.add_geometry(pcd)
    print("Load extrinsic:", param.extrinsic)
    ctr.convert_from_pinhole_camera_parameters(param)
    print("Post convert:", vis.get_view_control().convert_to_pinhole_camera_parameters().extrinsic)
    vis.run()
    vis.destroy_window()

Which yield the following:

Load extrinsic: [[ 0.73974973  0.12936515  0.66032946 -2.70437487]
 [-0.05661451 -0.96589918  0.25265306  1.57464615]
 [ 0.67049618 -0.22428427 -0.70719971  2.50352333]
 [ 0.          0.          0.          1.        ]]
Post convert: [[ 1.          0.          0.         -2.33576   ]
 [-0.         -1.         -0.          1.78849581]
 [-0.         -0.         -1.          3.9989113 ]
 [ 0.          0.          0.          1.        ]]

The extrinsics are properly loaded, but the ctr.convert_from_pinhole_camera_parameters(param) does not modify the view_control aspect of vis. Am I missing something here?

The above represents the simplest example I could find; however, I am not getting the expected behavior from customized_visualization.py, customized_visualization_key_action.py, and interactive_visualization.py.

Thank you!

Setup:
Python version: 3.9.7 | packaged by conda-forge | (default, Sep 29 2021, 19:23:11)
[GCC 9.4.0]
Open3D version: 0.17.0
Git tag: v0.17.0-1fix6008-23-g179886dfd

[saurabheights]

My 2 cents. This is one of the most common issue reported on open3d discord channel, since I joined there 3 months ago.

Some more info regarding this issue, during investigating other users complaint.

vis.get_view_control seems to be returning copy of view control, which is why making any changes doesnt help.

id(vis.get_view_control())
Out[14]: 140694699609776
id(vis.get_view_control())
Out[15]: 140697183871216

Here is an example, where changes made to view control doesnt propogate to Visualizer.

vis.get_view_control().get_field_of_view()
Out[7]: 60.0
vis_ctrl = vis.get_view_control()
vis_ctrl.get_field_of_view()
Out[9]: 60.0
vis_ctrl.change_field_of_view(step=1)
vis_ctrl.get_field_of_view()
Out[11]: 65.0
vis.get_view_control().get_field_of_view()
Out[12]: 60.0

I recently found this comment from 3 years ago where one can use ViewControl class/static methods that allows to change zoom value for ViewControl objects, but it didnt work for me as it should not since vis.get_view_control() returns copy.

[saurabheights]

I can look into how to fix this and PR, but first time working with PyBind and also diving into C++ side of open3d, so might take some time. @ssheorey - Is it ok if I take this up?

[ssheorey]

Hi @saurabheights thanks for offerring to help. Please try with the latest development Python wheel from here:
http://www.open3d.org/docs/latest/getting_started.html#development-version-pip
If the issue is still present, a good way to debug this would be go back in time for the relevant files and see which commit first caused the issue to appear.

[theNded]

Please feel free to assign me as the reviewer when the PR is ready. Thanks for your contribution!

[saurabheights]

@ssheorey Thanks for the tip. Helped in zoning in faster. The issue happened right after 0.16.0 and pretty sure (confirmed via building from source before and after the commit) it is coming from #2852. @pablospe Has located the issue as well. I will see if I can revert changes made by 2852 while keeping unproject functionality.

[natesimon]

Thank you all very much for your help in tackling this issue!

[saurabheights]

@natesimon My bad. Can you test with latest version from here ? My update to latest version got ignored (missed -U flag)

pip install -U --trusted-host www.open3d.org -f http://www.open3d.org/docs/latest/getting_started.html open3d

This issue was already fixed in #6116 and probably should work in your usecase. Do report back your result.

[natesimon]

@saurabheights (and others) Thank you so much for your help, this update worked for me!

The vis object now retains the correct extrinsic information. In addition, the other customized visualization scripts now also work. I am happy to close this issue.

Current output matches expected output:

Save extrinsic: [[ 0.68044456  0.18000249  0.71034802 -2.71285878]
 [-0.05642855 -0.95361307  0.29569905  1.50366424]
 [ 0.73062372 -0.24129072 -0.63872355  2.31622602]
 [ 0.          0.          0.          1.        ]]
Load extrinsic: [[ 0.68044456  0.18000249  0.71034802 -2.71285878]
 [-0.05642855 -0.95361307  0.29569905  1.50366424]
 [ 0.73062372 -0.24129072 -0.63872355  2.31622602]
 [ 0.          0.          0.          1.        ]]
Post convert: [[ 0.68044456  0.18000249  0.71034802 -2.71285878]
 [-0.05642855 -0.95361307  0.29569905  1.50366424]
 [ 0.73062372 -0.24129072 -0.63872355  2.31622602]
 [ 0.          0.          0.          1.        ]]

[theNded]

Thank you @natesimon and @saurabheights for the discussion and confirmation!
I will close this issue now. Please feel free to reopen if further problems occur.