Ability to setup custom CameraProjectionMatrix for Camera node and custom perspective vanishing point properties

[SirPigeonz]

Would be nice if user could setup custom projection matrix and had exposed vanishing point offsets for Camera node.
For example like in Unity: https://docs.unity3d.com/ScriptReference/Camera-projectionMatrix.html

Custom camera projection matrix is mostly useful for custom vanishing point for perspective (that is not in center of the screen), so exposed properties to setup this offset for perspective would be also great addition.

[BastiaanOlij]

I believe you can do part of what you are asking by playing around with the horizontal and vertical offset properties of the camera, though I am not entirely sure.

I did run into similar issues with processing projection matrices for stereoscopic rendering. The problem within Godot is that the camera is only loosely bound to the viewports and is therefor unaware of the size of the viewport. This means that you can't construct the projection matrix until you have identified both the viewport and the camera combination which is what happens within the render loop.

That said I am working on a change as part of the stereoscopic rendering that allows you to define a frustum to base the camera on. At rendering the frustum is adjusted by aspect ratio and the matrix created. This allows you to make asymmetrical projection matrices which should result in doing what you're asking for with changing the vanishing point.

[SirPigeonz]

I can't do it even partially, ^^' camera offsets work like extra method for moving camera in it's local orientation. Useful for camera shakes but vanishing lines are still placed in the middle of the screen and I need to have it moved to side so model on the side of the screen wont be stretched in ugly manner. ^^ (although I want it to be in extreme perspective, just on the side)

Cool! What you are working on sounds exactly as something I want to be able to do. :D Thanks! If you could expose properties to control these asymmetrical projections that would be great :)

[BastiaanOlij]

This is still based off of the master of Godot 2.2 and I spend until about 1am this morning on rejigging a bunch of the code I was doing but if you want you can give this a try:
https://github.com/BastiaanOlij/godot/tree/stereo_camera
I went to bed before giving it a good test and making sure there are no typos in the math so it may need some more work. One thing that is currently missing are the set/get left/right/top/bottom methods so you can't access the properties directly (Godot 2 doesn't have the new logic Reduz implemented for this in Godot 3)

It adds a new mode to the camera called Frustum and lets you set a left/right/top/bottom for the frustum. The size is at unit, not at the near plane, and the frustum should be sized ignoring aspect ratio of the viewport you're rendering on. Both are worked out later at render time.

There are also two new functions added to the camera, one that simply sets it in frustum mode and sets the given frustum, the other which calculates and sets the frustum mode for a given eye adjusting the frustum to a given IOD and convergence point.

If you set the camera up in Frustum mode for a given FOV you can do what you need to do simply by adding the same value to the right and left properties to start moving the center.

I'll eventually be updating this to Godot 3 but I'm still doing the openvr stuff in Godot 2 so not ready to make that move yet. See also PR #7121.

[seichter]

Was searching for this as well. Would enable off-axis rendering and properly setting camera intrinsics. Any work on this or should I make a PR? It's currently a deal-breaker for integrating vision based tracking and the like.

Update
Skimmed through the involved code. To do this in a sustainable way it will be far from trivial. It seems the actual View and Projection matrices (and their inverse) are cached in the driver code (a StateStruct to offload via Uniforms) of the Rasterizer. This needs a proper rework which someone, @reduz ? has to make a design decision. But I am rather astonished, as this is basic CG 101 to have developers access them directly.

[SirPigeonz]

This one "needs discussion" flag I suppose? :)

[rokasv]

I was just looking at changing the projection matrix manually myself only to find it's abstracted out.

I want to do some non-standard camera smoothing and without direct access, it's unnecessarily complicated, if not impossible, to do through the abstracted methods.

Please expose the camera transformation matrix to GDScipt.

[kjav]

I'd like to bump this issue as it is still relevant in 2019. Allowing direct access to the perspective matrix is a strength to the engine. For instance, I want to make a game which slowly transitions between an orthogonal and perspective transform, but I am unable to do so with the current setup. The current performance gain from caching the matrices could be maintained by having a 3rd "Perspective" enum entry, "custom matrix", or a new custom camera node.

[seichter]

I would argue that the current Camera implementation is certainly grown over time and would really need a rework. I tried to do it but its deeply entrenched with various indirections. There are multiple Camera classes involved handing over high-level information such as FoV etc pp. The actual projection matrix is computed just ahead of the rendering pass in the VisualServer, which I would say, is suboptimal. If the ProjectionMatrix is a first class citizen of the Camera in the scene all other variants of the Camera are actually much leaner.

[rsubtil]

I found this issue after asking on the forums, and I agree it's important to implement this. I'm currently using tricks with viewports and shaders to achieve a different view, but it's as simple as being able to change the shape of the camera.

[io12]

I also agree this is an important feature. I'm trying to implement a seamless portal Godot asset, and Oblique Near-Plane Clipping is necessary for the rendering to be correct. This requires a custom CameraProjectionMatrix.

[GameDevLlama]

This is still a thing in 2020 I guess? :-O I'm also currently struggling changing the camera's near plane to be custom.

[Calinou]

@SlothDevLlama You can use the Frustum camera mode with a custom frustum offset, which should cover some of the use cases of a custom camera projection matrix.

[GameDevLlama]

@Calinou thank you for your reply! Is there any image online showing how the offset actually works? Or what it does? As I understood, the offset shifts the frustum to the left / right - top / bottom relative to the camera's transform?
How can I achieve the near and far plane not to be parallel in the end?

Thanks!

[seichter]

@SlothDevLlama if you are planning to do something like off-axis stereo you are unfortunately out of luck with Godot. The dev's are rather unresponsive regarding the Camera implementation.

[Calinou]

@seichter There are improvements related to camera handling which are expected to land in 4.0 (especially for VR), but I can't make any guarantees.

[seichter]

@Calinou - that would be awesome. Can you point me towards pull requests that are solving this? I would love to see that! Would be great to replace the Unity and Unreal demos for our labs mini-CAVE with Godot.

[Calinou]

@seichter To my knowledge, there are no pull requests about this feature yet, sorry. Godot 4.0 isn't expected to be released before Q1 2021, so it may take a while.

[GameDevLlama]

@seichter luckily I'm currently experimenting with portals. But still the frustum mode of the camera is quite limited. I'd be better when developers were able to define the frustum by the 8 corner points or by defining a near and a far plane. Hopefully this will be possible in Godot 4.0

[Calinou]

@SlothDevLlama Indeed, reusing the built-in Plane type makes sense for this. There would have to be a way to define the plane size though, as the Plane type is designed to represent infinite planes.

[seichter]

@Calinou @SlothDevLlama I would argue that a 'Custom' mode with full access to the 4x4 projection matrix should be the base. All others are just parameterization variants on top.

[kjav]

@Calinou @SlothDevLlama I would argue that a 'Custom' mode with full access to the 4x4 projection matrix should be the base. All others are just parameterization variants on top.

I came here to say this and completely agree. This would allow custom wrapper nodes / resources on top which would solve everyone else's problems. This is the only solution which would allow full customisation of this rendering step.

I hope this is available in the next version of Godot, and while my c++ skills are not good enough to make this change I'd love to help test and document the feature.

[Kepler-Br]

I'm working with portal system too and this feature would be REALLY handy

[kjav]

Although I don't understand the implementation of this feature, it has been suggested that the current rendering engine makes implementing it almost impossible. Would it be prudent to treat this issue as an urgent feature to either say yes or no to in Godot 4, while the rendering engine is still in development? If not we might be in the same situation for a long time.

…

On Mon, 16 Mar 2020, 17:16 Igor Kosyanenko, ***@***.***> wrote: I'm working with portal system too and this feature would be REALLY handy — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#7499 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACE4QNV4HY3OXD6WWMVQK2DRHZNHFANCNFSM4C4AI67A> .

[seichter]

@kjav this is not a feature! I would argue this is a design flaw in Godot. I would love to see it being tackled with 4.0 right when the VK backend is worked on. Actually the current implementation limits the whole system in many ways. The intrinsic camera parameters are handed to the rendering backend (server) through a parameterized construct. There are multiple definitions of a Camera etc. pp. A good write-up about camera models and their implementation based on a 4x4 Matrix can be found here PBR Book - Projective Camera - it doesn't go into the details of other viewing frustums but with Vulkan (and any prior GL variant) the usual 4x4 model is used as well. Than why not use it (and expose it) in Godot? Currently I failed a few times to nudge the dev on this: the argument goes towards 'simplicity'. I don't buy into the whole 'simple for users' thing as it can be still exposed to the editor the exact same way as it is now. Just us XR people would have a much easier way (if there would be a proper access to the intrinsic) to integrate all kind of interesting rendering methods. I still hope we are being heard one day.

[reduz]

Hi guys, I would appreciate feedback of what would you use this for. Consider that things like portals or planar reflections will eventually be handled by the engine internally. Are there non low-level rendering use cases that need this feature?

[kjav]

Hi guys, I would appreciate feedback of what would you use this for. Consider that things like portals or planar reflections will eventually be handled by the engine internally. Are there non low-level rendering use cases that need this feature?

Hi @reduz , great to hear from you (I tweeted you earlier today with this link). My personal use case was a slow transition between an orthographic camera and a perspective camera. This was integral to the game idea. I tried using other methods, but I didn't feel I had the necessary control over the projection matrix to achieve it.

The camera's projection matrix is often one of the first things that a graphics course teaches you about, and one that can have a massive impact on how a game looks and feels. I think that exposing this low-level building block of games would allow a lot of creativity in the future that could be realised by writing your own plugins, building on this access to the projection matrix, instead of each new idea having to be implemented as a new camera setting.

Thanks for considering this!

[GameDevLlama]

Hi guys, I would appreciate feedback of what would you use this for. Consider that things like portals or planar reflections will eventually be handled by the engine internally. Are there non low-level rendering use cases that need this feature?

Hi @reduz nice to hear from you!
For it heavily depends on how the engine-implementation of the portals would work and how flexible it would be.

Sometimes you just want to cut your 3D world into slices that are not perpendicular to the camera view direction. I'm talking about non-euclidean games. When they're possible with the internal portal implementation it would be great.
But I can also think of some developers wanna have complex crosssections of the scene. (for technical illustrations of industrial machines for example).

[reduz]

@kjav well, you can kind of use a perspective projection with a a viewpoint further away to fake this effect, but letting this aside the main problem is that assuming things about the camera allows doing plenty of optimizations like fast z reconstruction from depth without relying on doing the full inverse multiply which is more expensive. Supplying a custom matrix will break this in a lot of cases, so all post processes and effects will screw up. Likewise code like the directional shadow splits are optimized and do a lot of assumptions based on the type of projection.

An alternative to this in 4.0 is to use the new global shader uniforms, so you could supply a custom projection matrix and use it in all your shaders directly, then do whathever you want in there.

[kjav]

As you've stated though, you can change properties of the camera such as viewpoint, which would surely change the projection matrix? What's the difference between changing the projection matrix in this way and just setting it to something each frame? (I'm not trying to be argumentative, just to understand!)

…

On Sun, 24 May 2020, 21:04 Juan Linietsky, ***@***.***> wrote: @kjav <https://github.com/kjav> well, you can kind of use a perspective projection with a a viewpoint further away to fake this effect, but letting this aside the main problem is that assuming things about the camera allows doing plenty of optimizations like fast z reconstruction from depth without relying on doing the full inverse multiply which is more expensive. Supplying a custom matrix will break this in a lot of cases, so all post processes and effects will screw up. Likewise code like the directional shadow splits are optimized and do a lot of assumptions based on the type of projection. An alternative to this in 4.0 is to use the new global shader uniforms, so you could supply a custom projection matrix and use it in all your shaders directly, then do whathever you want in there. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7499 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACE4QNXC3NT475YBURJI25DRTF4UXANCNFSM4C4AI67A> .

[seichter]

@reduz great to see you chipping in here. Seems I ignored the global shader uniforms for the wrong reasons ;)

My use case came from a simpler integration of AR or VR libraries where you are heavily dealing with custom projection matrices. Right now this is being handled by the rendering-backend which is fixed in the engine. But I would argue that any XR method would benefit from having a direct access to a projection matrix.

A gripe I have/had with the current implementation is that it is so non-standard that quite a few rendering and effect methods that are discussed in the CG community would have to be implemented in the backend or as you point out now with the global shaders uniform method. If it only can be done in the backend you basically need to modify the whole engine to get those things working instead of layering it on top of it. Something I would prefer.

My wishful thinking was, now the Vulkan based renderer is worked on, there could be a more modular camera system, like separating extrinsics and intrinsics of a camera (e.g. having a camera node that has two projections attached to it). Basically a layered system where at the core the projection matrix is stored and parameterizations (ortho, perspective, fisheye, pano ...) are synched with.

Naturally, if you change an over-constraint system to something minimally constraint, things will break. But I would guess there are also ways to solve it. Maybe this feature request is a sign that more people want to loosely tinker with the innards of Godot rather than have a fixed feature set.

Just my 2ct

[the-brickster]

I would like to add the use case that needs me to use the projection matrix. I'm currently porting over my code to godot that calculates the 2d bounding square of a 3d box using a method that is faster than just projecting all 8 corners of a box and taking the min/max of those projected points. The method needs to be done in the fastest amount of time possible, while there is the unproject_position method, I don't want to call unproject_position every single time for the potentially thousands or millions of 3d boxes I would be running the bounding square calculations on.

I hope to be able to calculate this technique using the complete 4x4 projection matrix without having to piece together a projection matrix. Would it be possible at a bare minimum to get access to a readonly version of the projection matrix that I would be able to do my calculations with?

[rsubtil]

My use case (a year back, and still is) is to be able to display a special orthogonal camera with a smaller size on the vertical than the horizontal (I explain it better here). For that I wanted to modify the camera's orthogonal shape to be a rectangle, and not a square.

I eventually found a workaround which is stretching the viewport's horizontal size until the vertical size matches the camera's total vertical size. This means that the viewport's horizontal size is, in my setup, 44% bigger than the actual screen's size, which means it's rendering quite more than the user can actually see; in our game, this shouldn't me much of a problem, but it's still a hacky workaround.

[akien-mga]

We're in the process of moving all feature proposals to the new dedicated repository: https://github.com/godotengine/godot-proposals

Would one of you be up for the task the open a proposal there following the issue template, and attempting to summarize the key takeaways from the discussion here?

[Calinou]

It's been 5 weeks since Akien posted a comment about reopening the issue on godot-proposals, so I'll close this.

---

Feature and improvement proposals for the Godot Engine are now being discussed and reviewed in a dedicated Godot Improvement Proposals (GIP) (godotengine/godot-proposals) issue tracker. The GIP tracker has a detailed issue template designed so that proposals include all the relevant information to start a productive discussion and help the community assess the validity of the proposal for the engine.

The main (godotengine/godot) tracker is now solely dedicated to bug reports and Pull Requests, enabling contributors to have a better focus on bug fixing work. Therefore, we are now closing all older feature proposals on the main issue tracker.

If you are interested in this feature proposal, please open a new proposal on the GIP tracker following the given issue template (after checking that it doesn't exist already). Be sure to reference this closed issue if it includes any relevant discussion (which you are also encouraged to summarize in the new proposal). Thanks in advance!