Add custom projection mode to Camera3D, make Camera3D store its CameraMatrix and expose CameraMatrix to user code

[Astrono2]

Describe the project you are working on

I'm working on a portal plugin.

Describe the problem or limitation you are having in your project

I need to use oblique frustum projection to allow for the portals' cameras to clip content that's behind the portal plane. Which is currently impossible in Godot.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

The enhancement would be to add a new projection mode, PROJECTION_CUSTOM to the Camera class, which would allow the developer to directly set the matrix to be used for projection. For my specific case, that would allow me to use oblique frustum projection. It would also address any other projection that any developer could want to use. And it would probably allow for some weird distortion effects as well.
Also, if a new projection is added with a CameraMatrix exposed to the user, it would make sense to migrate the entire Camera class and camera rendering to work with a CameraMatrix object belonging to each Camera instead of calculating the matrices from the Cameras' parameters.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

I think this can be done in a few steps:

1. Make CameraMatrix or other 4x4 matrix type available to user code
   1. Register CameraMatrix so that it is accessible to user code
   2. Add CameraMatrix to the documentation
2. Change the Camera class to allow for custom matrices
   1. Add a CameraMatrix object to the Camera class (could be named projection_matrix)
   2. Add updating the matrix when a parameter is changed (fov, near, far, etc.) by modifying the appropriate item in the matrix
   3. Add updating the matrix when the projection mode is changed by using the already existing methods in the CameraMatrix struct
   4. Add PROJECTION_CUSTOM to the Projection enum
   5. Add the CameraMatrix to the Camera's properties
3. Replace steps like this one in the rendering process with getting the camera's projection matrix.
4. Add necessary methods to either Camera3D or CameraMatrix to replace the functionality that calculating the matrix when necessary provided (e.g. setting the aspect ratio). So that the matrix doesn't have to be completely remade
5. Update Camera3D and possibly RenderingServer documentation
6. XR wizardry I cannot comprehend.

Here's a little mockup:

If this enhancement will not be used often, can it be worked around with a few lines of script?

Most definitely not

Is there a reason why this should be core and not an add-on in the asset library?

It requires considerable (although not very invasive) changes to two core classes and the rendering system.

[Calinou]

• Related to Add angled/oblique near and far camera planes #501
  and Allow for setting the X and Y FOV/frustrum sizes independently, and/or modifying the view transform #1250.

[lyuma]

Related to godotengine/godot#7499

My understanding is this feature is used for creating oblique frustums (mirrors / portals). Would a more limited feature like PROJECTION_OBLIQUE be sufficient to cover your proposed usecases?

[SaracenOne]

Yes, our specific use-case for this is creating portals/mirrors, particularly in VR. I would add also that the ability to seperate the view and clipping frustum seperately would likely go further in allowing optimisation, because we could limit the clipping plane to the confines of the portal.

[io12]

I was also working on a portal plugin and needed this (in addition to other things, like stencil buffering).

[elvisish]

I was also working on a portal plugin and needed this (in addition to other things, like stencil buffering).

It's one of the few things I've missed about switching to godot, having full access to the camera projection matrix. This portal plugin looks like it could be amazing if it could be completed with oblique frustum projection being available.

[Astrono2]

Related to godotengine/godot#7499

My understanding is this feature is used for creating oblique frustums (mirrors / portals). Would a more limited feature like PROJECTION_OBLIQUE be sufficient to cover your proposed usecases?

That's also the only use case I know, but in general I think having access to a Camera's matrix is a better solution. Specially since the changes I propose would also simplify adding new projection modes in the future. At the moment, the VisualServerScene::render_camera function scales with the amount of projection modes, as well as many of the functions in Camera.

CameraMatrix cm;

if (mode == PROJECTION_ORTHOGONAL) {
	cm.set_orthogonal(size, viewport_size.aspect(), p_z_depth, far, keep_aspect == KEEP_WIDTH);
} else {
	cm.set_perspective(fov, viewport_size.aspect(), p_z_depth, far, keep_aspect == KEEP_WIDTH);
}

This appears 3 times in camera.cpp. Storing the Camera's matrix in a variable would make the code a lot simpler.

[winston-yallow]

Exposing the possibility to set the matrix directly would be really nice in my opinion. It will also allow for more artistic effects like morphing between two matrices (even if not perspective/mathematically correct during morph you can achieve quite some interesting effects).

[wareya]

I want this to make it easier to fake 2d-style isometric and orthographic perspectives using 3d assets. Right now I need to stretch the world in bizarre ways to get the same effect, and it loses me a lot of control.

[Astrono2]

I left this hanging for a long time due to school. So I just edited to comply with not touching the Variant class and to update it for Godot 4, and I'll start working on it now!

[Calinou]

I left this hanging for a long time due to school. So I just edited to comply with not touching the Variant class and to update it for Godot 4, and I'll start working on it now!

Bump 🙂 Have you made progress so far?

[Astrono2]

I left this hanging for a long time due to school. So I just edited to comply with not touching the Variant class and to update it for Godot 4, and I'll start working on it now!

Bump 🙂 Have you made progress so far?

Sadly, no. I really want to work on this, but school is hitting hard. I think I'll get to it once I graduate at the end of the year. I have been reading the code and making minor changes to see what effects modifying this or that has, but not much more.

[qaptoR]

Hey all, I've just successfully implemented an Oblique Near Plane projection camera via the Godot source code.

Technically, I have Igor Kosyanenko to thank for doing the heavy lifting, specifically: figuring out how to apply Eric Lengyel's near clipping matrix transformation algorithm.

Igor originally posted his project status here:
https://godotengine.org/qa/64543/oblique-near-clipping-plane-for-camera?show=120466#c120466

And Eric's algorithm is found here (also linked in Igor's post):
http://terathon.com/code/oblique.html

None of Igor's project or source code is available anymore, so I had to make the adjustments to the engine (specifically: additions) adding the new camera type.

I have done my best to make the additions as minimal as possible (pretty much get the engine to compile, and render the proper oblique near plane).
I'm not sure yet what the appropriate channels are to go through to make a pull request (if it's even necessary), but I think it's at least ready for anyone looking for this type of camera to try it out in their projects.

The code can be found on my forked Godot repo on this branch:
https://github.com/qaptoR/godot/tree/3.4_oblique

let me know what you think!

Lastly, if you don't check out Igor's post: he specifically got stuck because his skybox was warping when seen through the portals camera view. The solution turned out to be setting a custom FOV for the skybox of whatever environment the portal camera can see.
The FOV should be the same as the portal camera (which should be the same as the player camera).

This is a short video showing the problem, and the solution:
https://youtu.be/hchttF-iN7Y

[qaptoR]

Hey. So, I just realized I dropped the code, but no explanation of the interface provided for the new camera. So here are some pictures:

This is the new option in the drop-down list

This is the new property, that can be manipulated in the 3d view editor
(thanks to Godot's great modularity)

This is the code attached to the portal's camera. The most relevant thing here is to make sure that the 'clip_plane' variable is the global_transform of the portal's mesh with the viewport_texture

These are some functions provided by Igor, that essentially replace the unity functions used by Sebastian Lague in his portal tutorial

Have fun!

[elvisish]

Hey all, I've just successfully implemented an Oblique Near Plane projection camera via the Godot source code.

Technically, I have Igor Kosyanenko to thank for doing the heavy lifting, specifically: figuring out how to apply Eric Lengyel's near clipping matrix transformation algorithm.

Igor originally posted his project status here: https://godotengine.org/qa/64543/oblique-near-clipping-plane-for-camera?show=120466#c120466

And Eric's algorithm is found here (also linked in Igor's post): http://terathon.com/code/oblique.html

None of Igor's project or source code is available anymore, so I had to make the adjustments to the engine (specifically: additions) adding the new camera type.

I have done my best to make the additions as minimal as possible (pretty much get the engine to compile, and render the proper oblique near plane). I'm not sure yet what the appropriate channels are to go through to make a pull request (if it's even necessary), but I think it's at least ready for anyone looking for this type of camera to try it out in their projects.

The code can be found on my forked Godot repo on this branch: https://github.com/qaptoR/godot/tree/3.4_oblique

let me know what you think!

Lastly, if you don't check out Igor's post: he specifically got stuck because his skybox was warping when seen through the portals camera view. The solution turned out to be setting a custom FOV for the skybox of whatever environment the portal camera can see. The FOV should be the same as the portal camera (which should be the same as the player camera).

This is a short video showing the problem, and the solution: https://youtu.be/hchttF-iN7Y

This is really amazing, I'll try it later, thanks so much for all of your work!

[qaptoR]

Thanks! I'm really grateful for everyone who took part in the discussion about this camera type over the years (which I came across in my research). All the back and forth between everyone really pointed me in the right direction of how to implement it.

This comment in particular pointed me to the right part of the engine to put the additions:
#501 (comment)

It really just goes to show how important documentation is. If anyone along the way hadn't bothered to post their progress, there's no telling where I'd be in the process right now.

[qaptoR]

I realized that the line: projection = PROJECTION_OBLIQUE
is redundant. It was a holdout from Igor's interpretation of Lague's approach. They switch back to a perspective camera when the portal camera in within a certain threshold... though I'm not sure why, leaving it as an oblique camera doesn't cause any issues.

Though they use two variables 'near_clip_offset' and 'near_clip_limit', the latter of which determines when the camera switches back to perspective (cam_space_dst <= near_clip_limit).
I didn't play around much with the near_clip_limit, but I did with the near_clip_offset: there's an issue when looking at the portals from extremely obtuse angles, where the camera can see where clipping plane cuts of the geometry.

I rationalized that this is why most portals are oval in shape, since the issue presents most prominently in the corners, and using a mask to cut them off would solve it.

I specifically want my portals to be rectangular, because I want them to give off that 'time-door' vibe from LOKI. So I applied a rectangular mask to the portal viewport mesh texture, and it nearly solves the problem, but since it's not ovoid, the corners still cause an issue. The second trick I've applied is deepening the frame around my portals, so that at extremely obtuse angles it's nearly impossible to see behind the clipped edges (except when I literally try to force drastically obtuse looking angles, which are unlikely to happen often/naturally, so at this point, I'm gucci w/ it.

However, I'd love to know if you also experience that issue, and what you decided to do to fix it in that event.

[Astrono2]

Hey all, I've just successfully implemented an Oblique Near Plane projection camera via the Godot source code.

Technically, I have Igor Kosyanenko to thank for doing the heavy lifting, specifically: figuring out how to apply Eric Lengyel's near clipping matrix transformation algorithm.

Igor originally posted his project status here: https://godotengine.org/qa/64543/oblique-near-clipping-plane-for-camera?show=120466#c120466

And Eric's algorithm is found here (also linked in Igor's post): http://terathon.com/code/oblique.html

None of Igor's project or source code is available anymore, so I had to make the adjustments to the engine (specifically: additions) adding the new camera type.

I have done my best to make the additions as minimal as possible (pretty much get the engine to compile, and render the proper oblique near plane). I'm not sure yet what the appropriate channels are to go through to make a pull request (if it's even necessary), but I think it's at least ready for anyone looking for this type of camera to try it out in their projects.

The code can be found on my forked Godot repo on this branch: https://github.com/qaptoR/godot/tree/3.4_oblique

let me know what you think!

Lastly, if you don't check out Igor's post: he specifically got stuck because his skybox was warping when seen through the portals camera view. The solution turned out to be setting a custom FOV for the skybox of whatever environment the portal camera can see. The FOV should be the same as the portal camera (which should be the same as the player camera).

This is a short video showing the problem, and the solution: https://youtu.be/hchttF-iN7Y

This is amazing! I still think giving the user access to the camera matrix is a good idea. But this covers my use case. I'll check it out when I have the time to get back to this!

[qaptoR]

Actually, now that you mention it...

I think it would be a good idea to transition the gdscript code for finding the clipping plane in camera space to the camera itself. That way, it would be easy to pass the camera the origin and normal of any plane, and get an oblique near clipping plane from that, rather than needing to attach a script every time (though you would still need a script to continuously update the values)

While I'm at it.. I'll make an attempt at exposing the CameraMatrix...
I think that will be slightly more challenging, since it would make sense to expose the whole CameraMatrix class to the editor, enabling us to directly access matrix manipulation functions without wrapping them into the Camera class.

Thoughts?

[qaptoR]

Okay, so I've moved the camera-space math into the set_oblique_plane function, and it works exactly as before (probably better performance).

Here's the code changed:

and what it looks like in the portal_camera script:

much, cleaner now; thank you for sparking the idea in my head to move the code for a sleeker interface.

It's not possible to overload gdscript bindings, so to keep both variations would require adjusting the naming to something like 'set_arbitrary_oblique_plane' for the new implementation, 'set_specific_oblique_plane' for the one using a quaternion.

I feel like there COULD be some unique use-case where the plane needs to be adjusted mathematically by more than just the offset provided by the arbitrary version... what do you all think?

EDIT: I decided to implement both using the exact names I mentioned.
commit 50b7ae: both implemented
commit 070f5e: only (pos/norm/offset) version implemented
commit 5eb5696: only (quat) version implemented

I think I'm going to hold off on exposing the matrix to the editor, since there are multiple ways to implement it... and I'm not sure which would be best:
expose the CameraMatrix class directly, create a new MatrixCamera node with an extended api, or simply extend the Camera api, both latter options wrapping CameraMatrix api functions to work directly on it's matrix (and figure out how to represent the 4x4 matrix in the editor).

[qaptoR]

Hey, I know I'm posting a lot here, I just know most of you are interested in implementing portals, which is my main reason for making additions to the engine (I need them too).

turns out that multiple setters was.. working somehow? but apparently also broken? I believe I may have just convinced myself it worked. In any case, godot does not like more than 1 parameter for setters, so I switched to a dictionary format for the oblique camera.

In addition to that, I've just finished implementing a new feature to the spatialmaterial, which will allow the specification of an arbitrary world plane to clip a mesh with.

Initially it was just a special shader code I found from a video:
godotengine/godot#3499 (comment)

and it worked well for basic uses, but I realized that when importing models with attached materials, I would have to convert those materials into shaders, and then inject the clipping code manually.. for every material on the model, and for every model (and that just won't cut it for me).

So after more deliberation than I care to admit (trying to figure out if I could use a 'next pass' shader to accomplish the same effect) I resolved to make an addition to the spatialmaterial, which can be toggled on and off.

The code for it is on the same branch as before, but here's a video showing both features in more detail (probably best to watch at 2x speed, i'm talking slower than it felt like).

https://youtu.be/zT6Zlx09UQY

[Astrono2]

I don't think modifying SpatialMaterial for this is a good idea. I toyed with the exact same idea, but my approach was to convert the spatial material to a shader and then inject the code. This approach doesn't require modifying the engine. (Also don't forget that engine changes should address specific things and make changes only for implementing those). This proposal is about giving users more control of the camera, any changes you want to make to SpatialMaterial should have their own proposal.

[qaptoR]

Sorry, I'm new to contributing to open source projects, and I understand that there's an etiquette I'm not yet aware of.

My last post was only to express a necessary fix to the oblique camera I hadn't realized was broken, and an additional change I had made to the engine on the same branch to the spatialmaterial. It was not a feature proposal, but for anyone who might pull from my branch might have noticed and wondered about why it was there (or possibly why it wasn't there in the official release).

I'm presently just starting the process of looking into how to go about setting up a pull-request formally for the oblique camera, there's a lot more documentation involved than I realized, so it won't happen immediately.
In the meantime, I will leave my 3.4_oblique branch as-is without further additions (or at least until I can figure out how to roll-back the spatialmaterial onto a separate branch).

On the spatialmaterial: it addressed my needs precisely, and is very useful for the 'passing through the portal' effect without needing to inject code into every material-made-a-shader manually.

I'm just enthusiastic about how much easier it is to improve the engine for my needs than I expected, but nothing I have presented thus far has been with the expectation that it would be integrated into the upstream repo.

[winston-yallow]

@qaptoR since you write that you are new, are you aware of the contributors chat? It's located at https://chat.godotengine.org/

There also is a #new-contributors channel over there in case you have questions about the PR workflow or related stuff.

[Astrono2]

Wow, okay. I sat down to get to work on this and the rendering system has gotten so much more complex since Godot 3. The handling and passing around of CameraMatrix objects is all over the place. This will take an enormous amount of reading code and drawing things out to even grasp what files are involved. I don't think I'll get it done by the release of Godot 4.

Sadly what was a non-trivial but doable change when I proposed it has grown in scope drastically. I think it would be best to close this with @qaptoR's eventual pull request. I still think storing a Camera3D's matrix as a property and using that for rendering is a good idea, but it's no longer a sensible solution to anyone's needs. It would be more of a structural change to allow for easier customization and more user control.

I will open another issue once I have a better idea of what storing a Camera3D's matrix and using it for rendering even means in the current rendering pipeline.

[qaptoR]

I'll be able to start working on the pull request after December 15th when finals are over, for anyone wondering why there is no progress on that in nearly 3 weeks, I was only able to tackle it over reading break.

[Power-Jake]

I done that in Godot 4

Hi, have you shared the code for this change anywhere? I would love to see it.

[DxUr]

I done that in Godot 4

Hi, have you shared the code for this change anywhere? I would love to see it.

Actually no but I will port this to 4.2 because it's done in 4.1 then I will make a pull request

[curtishimel]

DxUr, did that custom projections PR make it into the 4.2 beta that's currently in testing?

[DxUr]

DxUr, did that custom projections PR make it into the 4.2 beta that's currently in testing?

This is my implementation, idk if it added to 4.2 then I'll make a pull request later. I'm so lazy to do that.

[PitouGames]

I'm very interested in this feature, it will be really usefull to create application for VR CAVE.

I can't find any PR made by @DxUr. There is also no trace of custom projection in 4.2-beta1...

[mathiasmellemstuen]

I'm also interested in this feature when I need this for doing oblique projection planes. Currently working on a project with portals and non-euclidean worlds where modifying the projection matrix is a must.

[PitouGames]

I spent some time on this topic, because it's the base of my current project. I use Godot 4.1.2.

I think the editor is better like this:
Perspective

Custom

• I moved the property Current out of the way.
• When the projection mode is set to custom, the non-aplicable properties ("fov", "near", "far"...) are hidden.
• The default value for the custom matrix correspond to the default perspective camera on a squared window.

You can find my code here: https://github.com/PitouGames/godot/tree/custom_projection

• You can use set_projection(Camera3D.PROJECTION_CUSTOM) to switch the camera to the custom projection
• You can use set_custom(projection) to switch the camera and set the matrix at the same time. For exemple, calling set_custom(get_camera_projection()) will switch to the custom mode and keep the projection matrix that was used just before the switch. After that, it's your job to update the matrix, for instance when a user resize the window.
• You can use custom_projection = projection on a Camera3D node to update the custom projection matrix
• The default matrix value is the default perspective camera on a squared window.

Don't hesitate to give feedback!

[DxUr]

@mathiasmellemstuen
@PitouGames
the implementation on Godot 4.2 I think is different, this is why you can not find any PR by me.
then if you need it a lot tell me, I'll make it accessible from code NOT GUI, new people can break the projection and say GODOT is a bad blabla.

[mathiasmellemstuen]

@mathiasmellemstuen @PitouGames the implementation on Godot 4.2 I think is different, this is why you can not find any PR by me. then if you need it a lot tell me, I'll make it accessible from code NOT GUI, new people can break the projection and say GODOT is a bad blabla.

I totally agree with this. Making the projection matrix available only from code is probably the right way based on the reasoning you stated. Not sure how many will benefit from setting the coefficients manually in editor anyways.

Personally I would really appreciate if you go through with this.

[DxUr]

@mathiasmellemstuen ok, I'll do it, I hope they accept it for 4.2.

[Gamemap]

Would it be possible with the Camera Matrix to have an off-center perspective for a 3d camera?

Here is an example

[Calinou]

@mathiasmellemstuen ok, I'll do it, I hope they accept it for 4.2.

4.2 is in feature freeze, so any new features will have to target 4.3 at the earliest. You can still open a PR now, but it won't be merged for 4.2.

[DxUr]

@mathiasmellemstuen
@PitouGames
I do a PR for custom projection matrix implementation.
see godotengine/godot#84454

[DxUr]

Can any one tell the team to merge it, I really need it in my projects and compiling from source is waste of time.

[devalexx]

could we please merge the PR? from the code it's clear that the code doesn't break existing functionality, but bring the new one, which was requested already 3y ago and exists in unity for quite long time. or at least prepend '[experimental]' label on the dropdown

[Calinou]

could we please merge the PR? from the code it's clear that the code doesn't break existing functionality, but bring the new one, which was requested already 3y ago and exists in unity for quite long time. or at least prepend '[experimental]' label on the dropdown

Some issues were reported with that PR: godotengine/godot#84454 (comment)

Also, it still needs a review from a rendering maintainer before it can be merged.

[amonroejj]

My use case for custom projections in Camera3D is to orthogonally skew the whole world to give walls the appearance of height even when the camera is pointing straight down (mockup faked in Blender by moving the verts of the wall tops). This seems like a common use case for various kinds of 2.5 top down games.

Related technique is discussed on this page: https://manabreak.github.io/devlog/satos/3d/2019/08/16/3d-to-2d.html

[pineapplemachine]

I would really appreciate the option to experiment with different projections, e.g. spherical projection, without having to do weird workarounds in spatial shaders. What Godot projects are currently missing in 3D fidelity compared to other major engines can be made up for with art direction, but access to the projection matrix is going to be an important tool for creating a distinctive art direction. Is there anything those of us hoping for this feature can do to help move this forward?

[Calinou]

I would really appreciate the option to experiment with different projections, e.g. spherical projection, without having to do weird workarounds in spatial shaders.

Note that being able to adjust the projection matrix isn't sufficient on its own to implement things like panini projection. Alternative camera projections generally require a post-processing shader, or sometimes even stitching multiple camera renders together for particularly wide angles to avoid distortion.

[pineapplemachine]

Oh yes, many of the most interesting effects require more than just a custom camera projection matrix. But this is certainly one step along the way, and a helpful tool to have in any case.

[pyrareae]

This would be a greatly valuable feature for my use case. Lack of custom projection is a holdup for moving a concept I've been working on from Unity to Godot.

[periodyctom]

Now that 4.3 is out, how is it looking for getting this into 4.4?