Custom Camera3D projection: a practical approach

[Flarkk]

Describe the project you are working on

Godot

Describe the problem or limitation you are having in your project

There is a longstanding and strong push from the community to get fine control over the Camera3D projection.

The most plebiscited solution is to expose the camera’s Projection matrix to the end user (#2713, #4932).

While it remains a very valid target state, it comes with a number of yet unresolved problems :

• The projection matrix not only depends on the camera but also on the viewport’s aspect ratio to avoid undesirable stretching
• Many pieces of code across physics (picking) and rendering (including shaders) make strong assumptions on the frustum as optimizations, as it avoids full matrix multiplications : rectangular projection surface, parallel and z-orthogonal / z-symmetrical near and far planes, matrix inversibility

An arbitrary matrix may not meet these constraints and have visual and performance side effects. This is the main reason why no attempt has been successful so far (godotengine/godot#85529, godotengine/godot#84454).

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Give back full control solely over the parts of the matrix that really matter for an usage with a camera.

Expose it through an interface that guarantees the above constraints cannot be broken.

Benefits

• Almost straightforward to implement under the current status quo, unlike exposing the full raw matrix
• Still doesn’t restrict us in any way to move later towards exposing the full matrix
• Bring some foundations that may actually facilitate it … unless it makes us eventually realize that giving full access to the matrix is not needed after all

Use cases

Thorough reading through #2713, #501 and #3779 gives a pretty clear picture of what 💡 users need, what's ✅ already supported, and of what’s ❌ not needed :

	Symmetrical	XY panning	Shearing	Slanted near and far	Non-rectangular	Perspective division
Perspective	✅ Common use (current PERSPECTIVE mode)	✅ No use (can be achieved anyway by translating the camera)	✅ Doom-like (current FRUSTUM mode)	💡 Planar reflections (mirrors, portals)	❌ No use	✅ Across Z axis only
Orthographic	✅ Common use (current ORTHOGONAL mode)	✅ Already supported (only through rendering server)	💡 Oblique projections (cavalier, cabinet, etc...)	💡 Planar reflections and 2.5D	❌ No use	n/a

💡 Interpolation between perspective and orthographic
💡 Non-rectilinear projections (panini, fisheye, etc...)

Impact on matrix components

Above user needs are intentionally categorized to match the projection matrix' components.
This gives a clear picture of what really needs to be accessed :

Color coded	Analytical form
	With : $h=2\tan(fovy/2)$ projection plane height $n$ near distance $f$ far distance $\alpha$ aspect ratio $\lambda\in[0; 1]$ interpolation factor between perspective and orthographic $p_x, p_y$ panning values $s_x, s_y$ shear values $t_x, t_y$ slant values

💡 Access needed :

• ⚪ White : symmetrical
• 🟠 Amber : XY panning
• 🟣 Indigo : Shearing
• 🔵 Dark blue : Slanted projection plane
• 🔵 Light blue : Perspective division across Z vs orthographic

Light and dark blue require changes across rendering and picking code.
The rest only requires guardrails at input time regarding aspect ratio and inversibility.

❌ Access not needed :

• 🔴 Red : non rectangular projection plane
• 🔴 Pink : perspective division across XY

Giving access to these ones would have extensive impacts across rendering and physics code as it breaks the above assumptions.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

At server level

• Add function camera_set_projection() that lets the user define the projection in terms of [height, near, far, skew, slant, and persp/ortho mix]. This function enforces inversibility and does not constrain aspect ratio
• For compatibility and convenience, keep the current camera_set_projection_* but reimplement them by wrapping calls to the above
• Keep the current architecture : the server builds the matrix at each frame from the above user inputs. As an optimization, analytically build the inverse too, as well as the 6 frustum planes at each frame as those 2 representations are widely used across rendering code (today generated on the fly multiple times)
• Add additional code paths required to handle persp / ortho mixes across rendering and picking. This is the only pervasive change needed
• Do not branch on slanted projection planes. This is not optimal and has side effects, but acceptable for an auxiliary camera used for planar reflections only

At node level

• Camera3D attributes : height, fovy, near, far, persp/ortho mix (hidden by default), plus skew and slant sections :
  • Skew : a toggle bool and a Vector2
  • Slant : a drop-down to pick a mode (disabled, reflect, manual), a NodePath for reflect mode and a Vector2 for manual mode
  • Slant Reflect mode takes the path to a target Node3D to copy forward direction from (as proposed here). This is convenient for setting up reflections without having to adjust near and far manually
• Three presets PERSPECTIVE, ORTHO and CUSTOM enable/disable/override certain attributes to facilitate user input

Non-rectilinear projections

Projections like panini, fisheye and similar require additional changes to the rendering pipeline.
Typically a specific code path in the scene vertex shader is needed to perform a non linear projection before / after the projection matrix is applied.

This is a bit outside of the scope of this proposal, but for later reference :

• these code paths could be added as #ifdef sections, exactly like dual paraboloid rendering is handled today
• the rendering server could have specific methods to enable/disable these specific projection modes
• Camera3D interface could have additional presets like PANINI, PARABOLOID, FISHEYE and more

Exposing the current dual paraboloid mode to the end-user as an additional projection mode could be a first good place to start, as the bulk of the rendering code is already there.

XR support

Either left as is (the server bypasses entirely the user settings and retrieve the camera directly from the XR api) or reworked so that XRCamera3D extracts the camera attributes from the matrix provided by the API, then call the regular rendering server methods.

If this enhancement will not be used often, can it be worked around with a few lines of script?

No

Is there a reason why this should be core and not an add-on in the asset library?

It cannot be an add-on

[Calinou]

Typically a specific code path in the scene vertex shader is needed to perform a non linear projection before / after the projection matrix is applied.

Doing panini/fisheye with a vertex shader will have poor results on meshes that lack subdivision, so a per-pixel approach with oversampling is preferred here. It has a much greater performance cost, but it ensures you don't have to specifically build your assets for it. See also #3779.

[Flarkk]

You're right, I should have mentioned it.

At the time of writing this proposal I was thinking that some users might need the cheaper vertex-based implementation too (with assets tailored for this). As it is also much simpler to implement, I considered it a valid first step.

My intention is to primarily focus this proposal on rectilinear projections support, but of course any views welcome on this too.

[huwpascoe]

I'd like to get an intuition for this. Is there a demo or video somewhere that demonstrates these various effects, like sland and skew?

The doom frustum thing would also be good if anyone has an example. (I tried playing with the setting myself but don't get it)

[Flarkk]

Perspective

• Shearing (DOOM-like Y-Shearing effect) : see https://zdoom.org/wiki/Y-shearing
• Slanted near and far (mirrors / portals) : see Add new Camera3D type: Oblique godot#89140

Orthographic

• Shearing (cavalier and cabinet projections) : see https://en.wikipedia.org/wiki/Oblique_projection
• Slanted near and far (2.5D) : see Add custom projection mode to Camera3D, make Camera3D store its CameraMatrix and expose CameraMatrix to user code #2713 (comment) and Add custom projection mode to Camera3D, make Camera3D store its CameraMatrix and expose CameraMatrix to user code #2713 (comment)

---

I also quickly bundled a demo project that may help you getting a sense of the frustum's shape in each case.
Just open scene main.tscn, select node Frustum and play with the drop-down lists in the node properties panel.
drawfrustum.zip

	Symmetrical	Shearing	Slanted near and far
Perspective		Doom-like shearing	Mirrors, portals
Orthographic		Cabinet / Cavalier projections	2.5D

This is how the projection matrix is set in each case :

func _process(delta):
	if mode == Mode.Perspective :
		if effect == Effect.Symmetrical : matrix = Projection.create_perspective(30, 1.3, 3, 10)
		if effect == Effect.Shearing : matrix = create_frustum(30, 1.3, 3, 10, Vector2(0, -0.5))
		if effect == Effect.Slanted : matrix = apply_oblique_plane(Vector4(sqrt(0.5)/2, 0, sqrt(3.5)/2, 2.5), Projection.create_perspective(30, 1.3, 3, 10))
	if mode == Mode.Orthographic :
		matrix = Projection.create_orthogonal(-2, 2, -1.5, 1.5, 3, 10)
		if effect == Effect.Symmetrical : pass
		if effect == Effect.Shearing : matrix = apply_shear(Vector2(0, -0.5/3), matrix)
		if effect == Effect.Slanted : matrix = apply_oblique_plane(Vector4(sqrt(0.5)/2, 0, sqrt(3.5)/2, 5), matrix)
	

func create_frustum(p_fovy_degrees: float, p_aspect: float, p_z_near: float, p_z_far: float, p_shearing: Vector2, p_flip_fov: bool = false) :
	if p_flip_fov :
		p_fovy_degrees = Projection.get_fovy(p_fovy_degrees, 1.0 / p_aspect);

	var ymax = p_z_near * tan(deg_to_rad(p_fovy_degrees / 2.0))
	var xmax = ymax * p_aspect
	return Projection.create_frustum(-xmax + p_shearing.x, xmax + p_shearing.x, -ymax + p_shearing.y, ymax + p_shearing.y, p_z_near, p_z_far);

func apply_shear(shear: Vector2, matrix: Projection) :
	matrix[2][0] = shear.x
	matrix[2][1] = shear.y
	return matrix

# Borrowed from https://github.com/godotengine/godot/pull/89140
func apply_oblique_plane(p_oblique_plane: Vector4, matrix: Projection) :
	# Here goes oblique magic!
	# Eric Lengyel Solution: http://terathon.com/code/oblique.html
	var q : Vector4

	q.x = (sign(p_oblique_plane.x) + matrix[2][0]) / matrix[0][0]
	q.y = (sign(p_oblique_plane.y) + matrix[2][1]) / matrix[1][1]

	q.z = -1.0
	q.w = (1.0 + matrix[2][2]) / matrix[3][2]

	var c = p_oblique_plane * (2.0 / p_oblique_plane.dot(q))
	matrix[0][2] = c.x - matrix[0][3]
	matrix[1][2] = c.y - matrix[1][3]
	matrix[2][2] = c.z - matrix[2][3]
	matrix[3][2] = c.w - matrix[3][3]
	
	return matrix

[huwpascoe]

This is very helpful, thank you!

[huwpascoe]

I've managed to create the matrix, but there's one thing I can't work out. It's inverted on the z-axis compared to godot's own Projection.create_*() functions. I've tried negating different fields but there's not any difference. Any ideas?

Edit: I've discovered that changing the lambda is what's causing the inversion. The whole thing scales to nothing by the time lambda = 0.358 and reverses at 0.0.

Edit 2: The lerp formula doesn't work fundamentally I think. Ortho and Perspective will need to remain separate.

[ettiSurreal]

Would this allow for a reverse perspective effect (objects further away get bigger) like such?
https://www.youtube.com/watch?v=BRRridabnbc

[Flarkk]

@huwpascoe you messed up with parenthesis at various places in your code.
And I forgot a minus sign in the matrix in the issue description which inverted the frustum.

The updated code below works as expected.
You can test it out live in the updated version of my playground project (just pick Custom mode in the drop-down list and play with the sliders) :
drawfrustum.zip

var p := Projection.IDENTITY
var ln := lerpf(near, 1.0, lambda)
var h := 2.0 * tan(deg_to_rad(fov) / 2.0)
p[0].x = ln * 2.0 / (near * h * aspect)
p[0].y = 0.0
p[0].z = ln * slant.x / near
p[0].w = 0.0
p[1].x = 0.0
p[1].y = ln * 2.0 / (near * h)
p[1].z = ln * slant.y / near
p[1].w = 0.0
p[2].x = ln * shear.x / near
p[2].y = ln * shear.y / near
p[2].z = -lerpf(far + near, 2.0, lambda) / (far - near)
p[2].w = lambda - 1.0
p[3].x = ln * slant.x / near
p[3].y = ln * slant.y / near
p[3].z = -lerpf(2.0 * far * near, far + near, lambda) / (far - near)
p[3].w = lambda

There are probably a few details to iron out yet but it's good foundation you can build on I think.

[huwpascoe]

Great, it's working!

May as well make it feature complete.

@export var plane: NodePath
...
var obj := get_node_or_null(plane)
if obj is Node3D:
	var p := get_camera_transform().affine_inverse() * Plane(-obj.global_basis.z, obj.global_position)
	var v := Vector4(p.normal.x, p.normal.y, p.normal.z, -p.d)
	matrix = apply_oblique_plane(matrix, v)

I tried plugging a separate object into that apply_oblique_plane.

It....works? It's certainly tracking the plane. But it's very fragile, only certain angles and positions look like a frustum, everything else explodes.