What it takes to implement perspective correct texture mapping?

[tomekc]

I see it is on to-do list. I was looking for a possible way to implement, maybe even with some "cheating" like in Quake (affine interpolation every 16 pixels with correct Z).

My first guess is that Ebiten's drawTriangles is a limitation?

[SolarLune]

I believe the right way to solve this would be to involve the depth of the vertex in their corrected UV values; see this page on Wikipedia. I feel like this will need to be done on a per-pixel rasterization level (and so would necessitate changing the fragment shader Tetra3D uses), as well as us needing to send the depth of the relevant vertices along with the pixel information. This is essentially impossible currently, I think.

It might be possible if we use the depth texture to get the depth of the pixel.

[tomekc]

Right, I know the principles, but I am afraid it may be hard to achieve as it is.
I am not super experienced with shaders, but from what I see, ebiten internally provides only x,y coordinates to the shaders.

I was playing with modifying Ebiten so that Vertex coordinates are supplemented with z coordinate, but it takes some changes in the code (hardcoded offsets of fields in flat []float32 arrays). Unfortunately, I only got so far to see black screen (but no crash :) )

[SolarLune]

Yeah, adding in a Z coordinate would be very useful (or even just a customizeable amount of user-customizeable vertex data), though I feel like it should be possible to use the depth texture render result to get the relative depth of the pixel, and then, given the near and far ranges of the camera, get the relative depth of the pixel.

[tomekc]

I considered that direction too, but I have doubts.
By the way, what depth texture represents? If that's the one you can preview in demos via "depth debug" toggle, I have an impression, it is also incorrect, looks llike these are not true Z but linearly interpolated Z values across triangles.

[SolarLune]

The depth texture is the texture in the camera that shows how deep something when rendered through the camera. You are correct that it is the texture that you can see through depth debugging. The depth values are interpolated across the triangles, yeah (the depth is stored in the vertex color for the triangles when rendering to the depth texture).

That's the best method I came up with to actually render a depth texture (or something resembling it).

[tomekc]

I think the alternative way to go may be subdivision of meshes - like the PS1, which could only do affine texture mapping.
I did an experiment on the textured cube, and just adding one subdividion (so that one face is 8 triangles instead of 2), gave a pretty food result:

What do you think?

[SolarLune]

Subdivision does seem to be better, yeah. Currently, the Tetra3D Blender add-on has the ability to subdivide meshes before export, which works well enough (though is a bit buggy when it comes to triangles that join quad strips, for example).

I did some experimentation with dynamically subdividing triangles, but my method was to attempt subdividing the triangles after transforming them into view space. This didn't work (as you might expect, I suppose). I think it would work acceptably only if the triangle is subdivided prior to 3D transformation so that the new vertices get transformed to different locations.

[tomekc]

I think I see. I thought about dynamic tesselation. How hard can it be to implement?

There's a spot in camera.go commented with TODO: https://github.com/SolarLune/tetra3d/blob/main/camera.go#L1240

There is a simple algorithm that consistently converts one triangle into four:

[SolarLune]

I think I see. I thought about dynamic tesselation. How hard can it be to implement?

A little hard I think, haha.

Ideally, I wanted it to work without having to throw out the entire rendering process, but the current process renders triangles assuming purely the vertices contained within a mesh are used for rendering (as you might expect). Additional vertices created through tesselation need to be placed at the end of the vertex buffer so as not to interrupt other, non-tesselated triangles.

Determining which triangles should be tesselated is pretty easy; you can just check to see if any of the vertices are close enough to the camera. Beyond that, though, the extra vertex data needs to be generated and stored somewhere convenient, at which point they need to be transformed and lit, either separately from the mesh's native vertices, or together with them.

There is a simple algorithm that consistently converts one triangle into four:

That's the approach I took when I tried this out, and I believe it should work fine; it just needs to be done in 3D, rather than in 2D, before the triangles have been transformed.

[tomekc]

The problem caught me up and I wrote today a function which subdivides a mesh func (mesh *Mesh) Subdivide(), quadrupling the triangle count. For better result may be called many times :-) Results are not bad, but of course produces excessive amount of triangles (entire mesh gets subdivided).

Determining which triangles should be tesselated is pretty easy; you can just check to see if any of the vertices are close enough to the camera. Beyond that, though, the extra vertex data needs to be generated and stored somewhere convenient, at which point they need to be transformed and lit, either separately from the mesh's native vertices, or together with them.

Do you think it is doable with the current rendering pipeline?

For illustration, results of tesselation:

Original

Subdivided once (4x triangles)

Subdivided twice (16x triangles)

[SolarLune]

The results are definitely better in comparison, even with by a single subdivision.

Subdividing just the triangles that are the near the camera is the ideal, but it seems to be pretty complicated to implement, so a per-object approach like that might be easy as a short-term compromise. If a game is composed of larger pieces of individual scenery objects (like chunks of ground tiles), then each scenery object's fundamental meshes can be subdivided individually. Substituting the mesh based on distance is also useful for LOD, though it's not really necessary right now.

After thinking about this, my current idea is this:

1. Each mesh in Blender can be tagged as being auto-subdivided or not, and to what level (defaulting to the minimum, which is 1).
2. When you run a game and Tetra3D loads an auto-subdivision tagged mesh, Tetra3D will clone the mesh in memory and subdivide that clone using your function, storing it somewhere (probably as an extra pointer on the Mesh itself).
3. When a model that renders a tagged mesh is close enough to the camera, it silently switches from rendering the original mesh to the subdivided one. Otherwise, it uses the original mesh.

The upsides of this approach is that multiple models can use the same mesh, but only the ones close enough to the camera use the subdivided, higher-poly mesh.

Note, though, that this would break attempts to alter vertex attributes on meshes (like vertex colors), since the renderer switches between the normal, low-poly models, and the subdivided ones internally. I think documenting this would be important - an easy solution might be to simply add a function to update the mesh subdivision if you make any changes. Calling this would basically repeat step 2 above with the new mesh.

[tomekc]

Hello back @SolarLune ! I noticed that you recently started branch main...perspectiveCorrected. I see mostly a refactoring there, do you think of pursuing the way of custom shader?

I have been thinking about it too, but it would require passing additional uniforms to the shader, and rendering single triangle perspective correct. Too much to dive into existing codebase, hence progress is slow :)

[SolarLune]

Adding perspective correction to the existing shader Tetra uses (which is what I did in that branch) works pretty well, but without custom vertex attributes, we would have to pass the depth to perform the correction in a vertex color attribute (or possibly U or V). This means losing some color accuracy there (but not too much), if we were to, say, use half of the channel for depth, and the other half for the original color value. I put the depth in the vertex color's alpha attribute for that branch.

Triangles aren't properly clipped against the window, however, so the correct effect doesn't look right if the triangle is considerably off-screen.

[SolarLune]

OK, I added perspective-corrected texture mapping, but it's experimental; triangles aren't clipped properly at the borders of the screen, so you still get skewing if the triangle is massively offscreen. I'm also encoding the values necessary to perform the texture mapping in the alpha color channel of the vertices, which means that I don't have a lot of accuracy due to floating point errors, so the texturing is a little "fuzzy" (for now). This is most visible on animated textures.

Added in the commit: ec27c53