Firefly artifacts along seams between perfectly aligned meshes

[TokisanGames]

Godot version:
Master v3.2 abefd42
Probably other versions

OS/device including version:
Win10/64

Issue description:
In dark scenes, pixels from the background sky randomly, and inconsistently peek through the infinitely small spaces between two meshes, even when they are perfectly aligned (e.g. integer values w/ no rotation).

Discovered downstream in @Zylann's Voxel Tools: Zylann/godot_voxel#96

The voxel terrain now uses Transvoxel which aligns neighboring meshed sections to the same vertices. It should be seamless. However, the renderer flashes pixels from the background.

This is not limited to Zylann's project. In my minimal project below the issue appears with planes and cubes, even on perfectly integer-grid-aligned meshes using default materials.

It is subtle, unnoticeable in a lit scene, obvious in a dark cave. Here I took a video clip of 334 frames, while gently moving the camera. I merged all frames together with a 'lighter color' blend mode so you can see the dots add up. The dots actually follow a very thin line between the meshes but appear fatter due to my camera motion.

Notes from the VT ticket:

• Changing the background color/luminosity changes the fireflies color/luminosity
• The fireflies appear and change based upon camera movement, never in the same spot
• Fireflies also flicker on their own, per frame, without camera movement.
• We've experimented with the vertices in a vertex shader, scaling up blocks by a tiny amount, or growing them along normals, but none of that worked.

Steps to reproduce:

1. Turn down the lights, ambient energy: 0.1.
2. Have a bright sky background (brighten ground too).
3. Align cubes corner to corner, or planes, or any two meshes next to each other.
4. Move the camera around and look carefully at the seams between meshes.

Minimal reproduction project:

Here are 4 planes aligned with each other and two rectangular solids. The fireflies appear between all seams between objects. Between the planes, between the rectangular solids, and between the planes and rectangular solid. You can see the artifacts in the editor or by running the scene; vsync on or off.

firefly_bug.zip

External Reports & Workarounds
@Zylann

Last time I saw something like this in a 3D game, it was Minecraft https://www.reddit.com/r/Minecraft/comments/ek5vtv/white_dots_appearing_in_between_blocks/
And it looks like they still haven't solved it https://bugs.mojang.com/browse/MC-1794

Above, minecraft suggests adjusting mipmap settings (similar to Godot issue #25976? However this impacts objects with a plain white texture too), or tweaking your video card driver settings (Anisotropic Filtering, anti aliasing).

@Calinou

This could be worked around by darkening the sky when the player is detected as being in a cave. This is what Minetest does (it also suffers from this issue).
If you want to avoid affecting reflections, you could add a large transparent CubeMesh around the player that would effectively darken the sky (its cull mode will need to be set to Front for this to work).

@Calinou

Try changing the background mode to Keep in the Environment resource attached to the WorldEnvironment node. If it works, you should see a "hall of mirrors" effect when you move the camera in the 3D viewport. (PS: This is also the fastest background mode, so it should be preferred when working on a scene that's purely indoors.)

@Zylann

I tried this anti-firefly shader on a fullscreen ColorRect, it cleared all fireflies I was able to reproduce (will only work on isolated pixels). Of course, at a performance cost.
Zylann/godot_voxel#96 (comment)

[lawnjelly]

As I said on IRC yesterday, this is very probably floating point error in the pipeline. Your vertex shader has an input position which is multiplied by a transform matrix.

In math terms if you have 0.1 * 10.0, the result is exactly the same as 1.0 * 1.0, and the same as 100.0 * 0.01.

In floating point math this does not follow, float is only an approximation in many cases. 0.1 * 10.0 != 1.0. Float is 'wishy washy'.

So even in a situation where 2 blocks should line up exactly mathematically, unless the input is exactly the same, and the transform is exactly the same, they aren't guaranteed to line up.

There are a few ways of trying to bodge around this, such as having a slight epsilon on the scale of a block (with the scaling centred on the origin in model space, rather than a generalized scale, which will have no effect). And of course you can merge close verts in world space ahead of time if this is an option, but I'm lead to believe it isn't in this use case.

The way to guarantee the same result after transform, is to use the same transform for each block (i.e. world space to camera space etc, eliminate the model space from the transform). I.e. if you want to change the local translate of a block, do it before the transform matrix, and not as part of the transform matrix.

You can either do this prior to sending the vertex data, or you can use another stage (perhaps an integer translate) prior to the transform. However after this translate you should probably perform a rounding (NOT a floor) to ensure that any inaccuracy due to the translate is removed, prior to the transform (which may involve rotations). (the rounding ensures that the input is quantized to the same 'grid', but even this grid is not guaranteed to be exact integers because it will be stored as float on the GPU, however it is enough that it is repeatable)

Once rotations are involved, all bets are off in terms of rounding, so you want to make sure the input is quantized prior to the identical rotation.

You can test whether this is a godot issue or just a general transform math issue by writing some similar code in a simple opengl testbed. Maybe even shadertoy or something like that. Or just do the calculations manually will show they more often than not won't line up exactly unless the conditions I stated above.

This is all conjecture though .. If your input blocks are already in identical world space coords, and the transform is exactly the same, then something else is going on.

[TokisanGames]

Thanks for your message. I'm sure what you shared with Zylann was very useful for that case.

Your vertex shader has an input position which is multiplied by a transform matrix.

The example project I provided in this issue uses the default white material; no vertex shader.

Once rotations are involved, all bets are off in terms of rounding, so you want to make sure the input is quantized prior to the identical rotation.
You can test whether this is a godot issue or just a general transform math issue by writing some similar code in a simple opengl testbed.

In my provided example, all vertices are located at integer rounded positions, and the edges go along the global X/Y/Z axes. Is this an insufficient test?

I'm not sure what needs to be written in opengl to test whether this is a Godot issue beyond what I've already done in this ticket. Basic Godot primitives and materials exhibit the symptom, which is why I filed the bug here.

then something else is going on.

I think this is the case! :)

[lawnjelly]

I did a small manual example to show why this type of float error can occur:

	var pt = Vector3(1, 1, 1)
	var pt2 = Vector3(5, 5, 5)

	var trans = Transform()
	var trans2 = Transform()
	
	var tr = Transform()
	var tr2 = Transform()
	
	var axis = Vector3(1, 1, 1)
	axis = axis.normalized()
	
	trans.origin = Vector3(9,9,9)
	tr.basis = Basis(axis, 1)
	trans2.origin = Vector3(5,5,5)
	tr2.basis = Basis(axis, 1)

	tr = tr * trans
	tr2 = tr2 * trans2

	pt = tr.xform(pt)
	pt2 = tr2.xform(pt2)

	pt *= 10000000
	pt2 *= 10000000

Result : pt is 100000000, 100000000, 100000000
pt2 is 99999992, 99999992, 99999992

(actually they are both a little off from this due to the Godot debugger display, but you get the idea)

The general idea is that you are starting off with different points in local space, and you are relying on the transforms to line them up. And quite clearly, even though mathematically they should line up, they don't exactly, because of float error. The exact same thing happens in the graphics pipeline in the GPU, with the raw input data from the vertex buffer, and the transforms applied in the vertex shader.

If you aren't familiar with it already, I'd encourage doing reading on floating point error. It is one of the most common source of bugs in programming. Some people use fixed point / integers precisely to avoid this sort of thing. Another interesting alternative is the use of rational numbers.

[Zylann]

@lawnjelly you are showing this with extremely large values. This firefly issue has been seen in situations where none of the inputs exceeded the hundred, and were produced using integer math, even transforms with only integer translations. There aren't T-junctions either because Transvoxel is designed to deal with them. I'm aware of precision issues though, but then it would sound like we are just screwed.

[lawnjelly]

@lawnjelly you are working with extremely large values here. This firefly issue has been seen in situations where none of the inputs exceeded the hundred. I'm aware of precision issues though, but then it would sound like we are just screwed.

You get precision error with float with small values too. Try storing 0.1 in a 32 bit float, then read the actual value in a debugger. This is the nature of floats. They are 'floaty'! 😄

/edit. Incidentally, T junctions are another thing that can often cause issues such as this:
https://computergraphics.stackexchange.com/questions/1461/why-do-t-junctions-in-meshes-result-in-cracks

I didn't mention this originally because your example project only contained boxes (and not obvious T junctions), but it is worth double checking they are not occurring in the voxels. It is probably the transvoxel thing that is partly there to deal with T junctions.

T junctions as well as suffering from precision issues, also create cracks because of what I would call 'the fish eye problem'.

With a fish eye lens, straight edges in the world become curved edges through the view. If you view a box with a fish eye lens you see curved edges. And indeed this is what happens with a ray tracer. However, with 3d triangle hardware only the vertices are evaluated for position, and these edges are drawn as straight. If you then add another vertex in the middle of an edge (T junction) it will be evaluated in the correct curved position, away from the straight edge.

[blockspacer]

If it is project issue than how it may be fixed? (i`m about minimal reproduction project with 4 planes aligned with each other and two rectangular solids)

[blockspacer]

@lawnjelly
You said about floating point errors
I can`t find any places where floating point errors may happen in minimal reproduction project with 4 planes aligned with each other and two rectangular solids, no vertex shader.
Is it godot related issue or something wrong in provided example project?

[KoBeWi]

I can see the glitch in 3.2.4 beta4, but not in 4.0, so this is fixed.

[Zylann]

Given the nature of the issue I doubt this was fixed, at least not intentionally. Maybe you were lucky, maybe it actually still happens... we'll see.

[KoBeWi]

You can try with a nightly build.

[TokisanGames]

This bug has definitely not been fixed and should be reopened.

v4 changed the environment, which caused this project to not be set up properly to highlight the bug. But it is most definitely there, visible under the right conditions. That being dark meshes in front of a light sky or other background.

Here I've set up a project with a new environment. The meshes haven't moved. Move the editor camera around where the planes and cubes meet, and the fireflies become obvious. There are no materials, just low light, with a bright background. Doesn't matter if there are cubes meeting corner to corner, or planes meeting cubes, or planes meeting planes, parallel or perpendicular. Doesn't matter if it's a procedural sky or a color as is used here.

firefly_bug_v4.zip

16524d4
Sat Dec 19 09:34:41 2020

[FilipLundby]

Setting the MSAA to 8x-16x minimized the issue, but of course that's really performance heavy.

v3.4.beta4

[Calinou]

Would centroid sampling be able to alleviate this? I don't know if it's available in GLES3 though. If it's likely limited to desktop OpenGL only, would have to be conditionally added on shaders on desktop platforms only (which would add a fair amount of complexity).

Centroid sampling is definitely not available in OpenGL 2.x/GLES2.

[TokisanGames]

https://stackoverflow.com/questions/39958039/where-do-pixel-gaps-come-from-in-opengl

There are solutions on this page from scaling by 1.00001, disabling greedy meshing, etc. However in this link it only occurs with MSAA off. In Godot, it also happens when on.

They say

OpenGL (and all other hardware rasterizers) only guarantees gapless rendering of the edge between two triangles if the edges exactly match. And that means you can't just have one triangle next to the edge of another. The two triangles must have identical vertices on the shared edge between them.

So gapless rendering is already provided by opengl, but perhaps Godot is not meeting the requirementsfor using it:

The other thing you have to do is make certain that the two shared vertices between the two triangles are binary identical. The gl_Position output from the vertex shader needs to be the exact same value. So if you're computing the position of the cube's vertices in the VS, you need to do that in a way that will guarantee binary identical results.

In my minimal projects above, Zylann's voxel terrain, and hterrain the fireflies are present, even under MSAA. Yet the vertices of adjoining meshes are at the same integer location, with perfect matching edges. Yet there are gaps. This suggests that the above mentioned caveat is not happening with the engine VS shader.

Here's a discussion at Khronos with the same issue, with a possible solution regarding gl_position and the centroid qualifier. ???

https://community.khronos.org/t/issues-with-triangle-seams/106969/7

[clayjohn]

@tinmanjuggernaut I took a look at the links you posted and then ran the MRP for myself. Like @lawnjelly and the person in the stackoverflow article I felt that the issue was likely to do with floating point precision in the vertex shader. To test that theory I added the following shader to the planes in the MRP:

shader_type spatial;
render_mode  skip_vertex_transform;

void vertex() {
	VERTEX = (MODELVIEW_MATRIX * vec4(VERTEX, 1.0)).xyz;
	VERTEX = round(VERTEX*100.0)/100.0;
}

This rounds the vertex position to the nearest hundredth after applying the transform. Running the MRP using this shader removes the artifact completely on my device (it also results in vertices being locked into nearest hundredth precision which may not be desirable).

Remember, each object has a different transform, and both vertex positions and transforms are specified in floating point values (even if you use whole numbers in the editor). So even if the origin of the transform added to the local vertex position should equal, the number of transforms applied create enough floating point precision error that the vertices end up slightly offset.

The ideal solution is outlined nicely in lawnjelly's answer above. Alternatively, you can add a vertex shader hack like I have here (but maybe use more than hundredths precision)

[TokisanGames]

@clayjohn
In 3.4.2, using the minimum project, Cube, Cube2, plane3 and plane4 have no rotation, yet the fireflies appear where they connect. Cube2 and plane3/4 edges are different sizes, so as my links express, they cannot guarantee gapless rendering. However cube/cube2 and where the planes meet, they have perfect edges so should be gapless.

I applied your vertex shader to the material of all objects, and indeed no longer see fireflies between cube and cube2, but it exacerbates the one between cube and plane3/4, which do not have identical edges.

With the vertex shader, I also see the fireflies between the planes, which do have identical edges, though not as pronounced or as stable as shown here.

Regarding the precision amount, I used 10/10 which made all camera movement quite steppy as the vertices snapped around. Then I switched to 1000/1000 and they still appear, though less frequent.

I also applied this to @Zylann 's terrain, which should have perfectly matched edges, and it made the fireflies much worse, and growing the vertices by even 0.1 also made it worse.

Regarding lawnjelly's solutions you mentioned, they are:

1. the vertex shader has an input position multiplied by a transform matrix, producing floating point error.

As previously discussed, in the MRP we're using Godot's standard material shader on Godot's standard mesh instance primitives with integer based positions and sizes, axis aligned with no rotation on many objects with fireflies at seams. There is no floating point error or matrix transforms applied in user space. This is an engine problem, perhaps in the standard shader.

2. apply a scale to each object

I applied Godot's vertex scaling method VERTEX+=NORMAL*grow;, and while this does help the cubes, no amount of scale fixed where the planes met. Even 1, which should have had adjacent planes w/ the same rotation overlapping significantly, still produced fireflies between them.

3. merge close verts in world space

This is out of scope for user space using imported meshes and native shapes. Perhaps it's something you can do in the engine.

4. use the same transform for each block, eliminate the model space from the transform -

The same response as above. The MRP shows fireflies between objects with identical edges, integer vertices, and no user applied rotations.

Calinou suggested something and the links I included might be ways to fix the problem in the engine, but it's out of scope for the user since we have limited access to the pipeline.

[clayjohn]

@tinmanjuggernaut

I see you are still repeating a few misunderstandings. When the stackoverflow article and lawnjelly refer to perfectly matched edges they are talking about perfectly matched edges before the transformation matrix is applied not after. In the MRP you have aligned planes so that they share an edge, but this is not the same as having perfectly matched edges. In order to have perfectly matched edges you need to be using the exact same transformation matrix (so no translation, scaling, or rotation) and local vertex positions. In other words, the MRP does not have perfectly matched edges because the local vertex positions are different, you are relying on the transformation matrix to transform them into the same position.

we're using Godot's standard material shader on Godot's standard mesh instance primitives with integer based positions and sizes

GPU's don't deal with integer-based positions and sizes, everything remains in floating point. While an addition between two whole numbers in floating point shouldn't create any risk of floating point error, we aren't just adding numbers together in the vertex shader, you are first transforming from local vertex space into view space (camera space). The transformation into camera space includes the position, rotation, and scale of the camera, which in the MRP is not using whole numbers. Further, vertex shaders (in OpenGL) expect their output in clip space, not world space. essentially that means that the positions are translated into the [-1,1] range. As soon as that happens you introduce floating point error as well. Note: this is not a quirk about Godot, this is just how modern graphics APIs work. The way around this problem is described by lawnjelly above:

The way to guarantee the same result after transform, is to use the same transform for each block (i.e. world space to camera space etc, eliminate the model space from the transform). I.e. if you want to change the local translate of a block, do it before the transform matrix, and not as part of the transform matrix.

Remember, the GPU takes vertex positions in and transforms them by the modelview (local to world space and world to camera space are combined into one operation) matrix and then converts them to clip space with the perspective matrix. Despite the user facing transform only using whole floats and the edges seemingly aligning, the camera-space transform introduces non-whole floats in the local-to-view space transformation. What lawnjelly is suggesting above is separating out the different transforms. With your MRP, in theory, if you separated out the local-to-world transform, you should end up with identical positions in world space, meaning the floating point error introduced in the world-to-view and view-to-clip transformations should match for the vertices. This approach would not work if you rotated any models though.

edit: And here is the shader code for what he describes, Note: this only works if everything uses whole numbers and there are no rotations in the transformation matrix:

shader_type spatial;
render_mode  skip_vertex_transform;

void vertex() {
	VERTEX = (WORLD_MATRIX * vec4(VERTEX, 1.0)).xyz;
	VERTEX = (INV_CAMERA_MATRIX * vec4(VERTEX, 1.0)).xyz;
}

The core of the problem here is that you are modifying the transformation matrix to line up the edges of vertices exactly and expecting that the GPU's rasterizer will output perfectly aligned sets of pixels. In general it is bad to assume that the GPU will do anything with perfect precision (especially when using OpenGL). GPU's are designed with a bundle of heuristics and simplifications in order to keep them fast. As a result, they are fast but inexact. As a user designing content for consumption by modern GPUs, you need to be aware of this inherent limitation and design your assets accordingly. Within a single mesh, you just have to ensure that the triangles have perfectly matched edges to ensure that this form of artifact doesn't arise. But when using multiple meshes, the easiest solution is to overlap them just a tiny bit.

edit: I forgot to cover one more thing

I applied Godot's vertex scaling method VERTEX+=NORMAL*grow;, and while this does help the cubes, no amount of scale fixed where the planes met. Even 1, which should have had adjacent planes w/ the same rotation overlapping significantly, still produced fireflies between them.

This is not the same as vertex scaling. This offsets the vertices by the normal. When lawnjelly and I talk about scaling we are talking about the scale property of spatial item's transform which actually makes the mesh a little bit larger so that vertices overlap instead of align. Alternatively, you can add VERTEX *= scale_amount; (note the multiplication rather than addition). The code you used keeps the vertices nearly aligned but would introduce more floating point error.

[TokisanGames]

Thank you @clayjohn for these solutions, and for explaining what I did not understand from @lawnjelly. They will all work.

At the end of the day, "the reasons" won't matter to the average end-user. On this ticket, neither I nor Zylann were able to use the information until you spelled it out for me. It still seems to me to be a bug in the renderer, but having these workarounds is a second best alternative. Here's what I found with them:

1. Manually overlapping meshes

Sure it can work for regular meshes being placed by a level designer. But not good when constructing an arraymesh by code for a terrain or voxel object/terrain where the seams need to be perfect at every viewable angle, or mixing multiple LODs. This method may produce gaps or be difficult to implement automatically.

2. Apply vertex or object scaling

This works fine on the MRP and simpler objects. Either enter an object scaling of say 1.001 on the transform properties in the inspector, or add something like this to the shader, then increase vertex_scale in the shader parameters until the problem goes away, or hardcode 1.001, etc.

uniform float vertex_scale = 1.0;

void vertex() {
	VERTEX *= vertex_scale;
	...
}

This somewhat worked on @Zylann 's hterrain, however since it uses LODs, getting the right value for LOD0 created gaps on higher LODs. I created a sophisticated version of this to vary the vertex scale based upon distance and had a lot of trouble getting the right numbers for every possible case to eliminate both fireflies and gaps, though I got about 97-99%.

3. Separately transform the vertices through world space, then view space.

This worked perfectly for zylann's hterrain and voxel terrain. I tried manually converting directly to view space through MODELVIEW_MATRIX, but this produced fireflies and must be the default method used in the engine. When I went back to the two separate transformations that eliminated the fireflies.

shader_type spatial;
render_mode  skip_vertex_transform;

void vertex() {
	VERTEX = (WORLD_MATRIX * vec4(VERTEX, 1.0)).xyz;
	VERTEX = (INV_CAMERA_MATRIX * vec4(VERTEX, 1.0)).xyz;
}

I'm satisfied enough to close the ticket. Anyone want to keep it open to track possible changes to the engine?

@Zylann For hterrain, the vertex lines go at the end of the function to eliminate the problem.

For voxelterrain, while it does work to fix the meshes, skip_vertex_transform messes up the normals for both of our triplanar algorithms (yours and mine/godot's). We can look at this on the voxel terrain ticket.

[elvisish]

I'm using a shader that darkens walls and the artifacts are especially obvious:

2023-06-29.22-49-10.mp4

I tried both shader suggestions and neither of them do anything to fix it unfortunately.