Proposed: Order Independent Transparency

[bhouston]

Description of the problem

We (@Jozain and myself) are currently implementing out-of-order transparency for ThreeJS for a project of ours. We'd like to contribute it back. We are implementing the method of Morgan Mcguire, Weighted-Blended Order Independent Transparency: http://casual-effects.blogspot.ca/2014/03/weighted-blended-order-independent.html

We are helped a lot by @arose's example OIT code on which our stuff will be partially based, just our stuff will be designed into the core of Three.JS: #4814

Here is an proposed example of how to enable this mode:

var renderer = new THREE.WebGLRendererer( .... );

// add new "transparency" variable to renderer to select the transparency rendering mode
//renderer.transparency = THREE.PaintersTransparency; // the current ThreeJS method, the painter's algorithm
renderer.transparency = THREE.OrderIndependentTransperancy; // the new OIT method

This mode will be implemented by a new WebGLOrderIndependentTransparency class that will be responsible for managing the buffers. It will create two additional RenderTargets that will track the size of the current render buffer. The first will be the accumulation buffer which we will render all transparency objects to and then we will render the objects as well to some alpha product buffers - thus two separate renders. This will be followed by a "resolve" stage that renders the transparent objects over top of the existing beauty render of the opaque objects.

This workflow will work with base WebGL and work with multi-sample buffers as well. It can obviously be speed up using multiple render targets, but this shouldn't be that slow for non-huge numbers of transparent objects.

We are going to implement this now, just wanted to give a heads up as to our design so that if you have feedback on it, we can incorporate it now.

/ping @WestLangley @arose @spidersharma03

[tentone]

This would be really nice, since it seems like threejs will be moving for a new renderer for WebGL2 it would be a good chance to take a look at this and this (#9620)

[bhouston]

http://Clara.io will be staying on the current ThreeJS WebGLRenderer until WebGL 2 is available on more than 80% of devices - thus it needs to be available in Edge, Firefox and Safari before we can consider moving over to it.

[NeilGatenby]

Hi. You guys seen http://graphics.cs.williams.edu/papers/TransparencyTVCG17/? Same guy, more recent work? It's all about fog and dispersion and refraction and stuff which this issue is NOT about, but equation (5) is (I think) an improvement on equation 10 from the 2013 work that you guys are looking at.

https://www.youtube.com/watch?v=jWe5Ae22Ffs#t=8m59s, too

Hope this helps

Neil

[NeilGatenby]

So I got it working using blended transparency as per eqn(5) of http://jcgt.org/published/0002/02/09/. The @arose demo was a God-send! Thanks. Note, I've NOT implemented weighted blended transparency - for the reasons I outline here, for the interest of others:

(1) The weighting function is hard to tune. @arose has noted this, the paper itself notes this (mentioning at least SEVEN different functions), and unless all my scenes are very similar, that is a problem

(2) As noted in the 2013 JGT paper, the transparencies change as you get nearer to (further from) your scene! They say "depth-weighted results are surface order-independent but not translation-invariant along the depth axis". It's highly annoying when you moving through the scene!

(3) They justify the "weighted" part of their paper with lots of examples. Comparing their "blended" images to their "weighted-&-blended" images, one is hard pressed to spot the difference!!

Their (non-weighted) blended approach is better that Bavoil and Myers 2008's blending function, which is in turn better than Meshkin's 2007 original (see JCGT link, above, for specifics on those papers)

Hope this helps

Neil

[arose]

thanks @NeilGatenby, do you have an example online by any chance?

[NeilGatenby]

No

But am happy to sort that ... will post here when done

Still struggling to ...
(1) get the opaque render behind/amongst the transparent and
(2) initialise the revealage texture to 1.0 (or 1,1,1,1) ... I figure the accumulation texture is auto-zeroed, so less of an issue. Since my images (of just transparent objects) look good, I'm curious to see what will result when the revealage texture is properly initialised

N

[NeilGatenby]

dev...NeilGatenby:oit is the diff between between my OIT and @arose's OIT ... hope that helps, all

@dunkyp deserves an honourable mention for any progress I made here - he helped me get started on this

[WestLangley]

@NeilGatenby NIce demo. IMHO, I'd set

transparentMaterial.depthWrite = false;

for a fair comparison with oit.

[NeilGatenby]

Thanks. Certainly improves that case. I guess things will go (more) awry when the transparent meshes are mixed up with opaque meshes that I want depth writing ON for?

[WestLangley]

Not necessarily. Opaque materials are rendered first by the renderer; transparent materials second. If the user sets material.depthWrite = false for the transparent materials, it can work fairly well, as you can see.

It can be problematic if half-semi-transparent/half-opaque textures are involved, however.

[NeilGatenby]

Interesting stuff. Thanks again. Sounds like it might work well for an "in between" quality ... when you don't want to wait for OIT or suffer the painter artefacts I'm seeing now.

Currently I can't get OIT working with opaque, anyway ... still struggling with the points I mentioned in my first post of today, viz... I want the depth buffer from my opaque pass to prevent both transparent passes from working where they needn't. Then I want to combine the colour buffer from that same opaque render with the 2 transparent textures I've created, for the final image.

Sounds easy in software but I'm struggling to do it with my limited three.js skills! ☺

[Ben-Mack]

@bhouston Can you share the code for Order Independent Transparency that you've implemented?

[bhouston]

The best code is actually @arose's code here: #4814

[trusktr]

Looks like @pailhead has also made some progress on order-independent-transparency here: https://discourse.threejs.org/t/depth-peel-and-transparency/5365 (or is that something else? It seems to solve some issues that I've seen with the order-dependent transparency)

[hrgdavor]

any progress on this ?
is there a relatively new version of thrrejs fork where this somewhat works ?

[mrdoob]

@hrgdavor No progress.

[pailhead]

Not sure if this is necessary, but i found it useful for the topic. I've tried to since clean the implementation, but i was never able to figure out if the buffer limitation can be worked around. FWIW:
#15490

[bhouston]

Myself and https://github.com/Threekit are pleased to put a bounty of $500 USD on an accepted PR of an implementation that supports order independent blending of transparent surfaces using the floating point buffer blending technique described in the original issue description above (or any further improved version.)

[stevinz]

I have implemented weighted, blended order independent transparency as a Pass that can be used as a stand-alone renderer, or in combination with other passes within an Effect Composer composition.

Github Repo - Online Example

This example builds on the progress made by @arose, @pailhead, and others.

Instead of monkey patching existing shaders (like 24227), the pass includes a material MeshWboitMaterial. This material is functionally equivalent to MeshBasicMaterial, with the addition of a weight property. This is implemented in the style presented in MeshGouraudMaterial.

This pass doesn't need access to internal renderer framebuffers, and does not create a depth texture. In doing so it remains WebGL 1 compatible and mobile friendly (although, iPhone still doesn't support writing to gl.FLOAT framebuffers and falls back to gl.UNSIGNED_BYTE which works, but doesn't look as nice).

It is able to avoid creating a depth texture and the need to bind multiple render targets by re-using the depth buffer of the current writebuffer after each step. It also is able to generate it's own list of opaque / transparent objects and render them separately by implementing a visibility cache (similar to OutlinePass).

I would be happy to submit a pull request to add this into the examples if this approach looks acceptable.

[LeviPesin]

Wouldn't it be better to integrate this in the core (by creating shaders for WBOIT and adding more code to the renderer) instead of creating a new material and pass?

[donmccurdy]

It would probably be too soon to bring into core. There are going to be some limitations to the approach described. I would start out separate, see where/when this is helpful to users among the different transparency options, and go from there.

I'd like to try it out though, and would be interested in an example!

[stevinz]

The advantages of being in the core are higher performance and built-in shader support.

One of the biggest benefits of OIT is not having to sort large numbers of transparent objects. This method can achieve similar performance to that of being in the core with the addition of a way to overwrite / disable transparent object sorting (#24809, #19305).

And it would be helpful to have support in all the standard shaders. Support can be added to any shader, though, by monkey patching with onBeforeCompile. The idea of including MeshWboitMaterial was to have out-of-the-box support for the technique while also serving as a helpful example of how to implement it into a shader.

[pailhead]

The advantages of being in the core are higher performance…

This is a theorem, not an axiom. Things have been implemented in the past that blow the performance of built in things.

My two cents are - it’s great to have examples that modify shaders!

This is because people are interested in modifying shaders, so I assume that they would like to see examples. I myself have almost exclusively looked at examples that modify shaders over the past ten years.

Another benefit is that people have visibility into certain approaches/algorithms. My OIT proposal has been sitting here and on the forum for years. Some people stumble upon it sooner, some later, some never. It would be perhaps considerate to those people to give them various assorted examples, as opposed to hiding them based on some obscure policy.

onBeforeCompile basically legitimized any and all monkey patching when it was introduced into threejs. I don’t see any sense in trying to prevent people from using it.

Also, this being an example means - just like you maintainers don’t have to look at millions of peoples apps when you do an update to see if their apps break, you also don’t have to ensure that examples don’t break.

The way this team treats examples is the problem here, not the common people who want OIT - a technique that’s been seemingly around before the programmable pipeline.

[bhouston]

We implemented this in our private fork of Three.js in 2016 with defines within the main materials using this approach:

oit_pars_fragment.glsl

#ifdef USE_OIT

  varying float vPositionZ;
  uniform int oitMode;

#endif

oit_fragment.glsl

#ifdef USE_OIT

    if( oitMode == 0 ) { // Accumulation Pass
      float z = abs(vPositionZ);
      //float weight = max((min(1.0, max(max(gl_FragColor.r, gl_FragColor.g), gl_FragColor.b) * gl_FragColor.a)), gl_FragColor.a) * clamp(0.03 / (1e-5 + pow(z / 200.0, 4.0)), 1e-2, 3e3);
      //float weight = pow( gl_FragColor.a, 1.0 ) * clamp( 10.0 / ( 1e-5 + pow( abs( z ) / 5.0, 1.0 ) + pow( abs( z ) / 200.0, 1.0 ) ), 1e-2, 3e3 );
      float weight = gl_FragColor.a * clamp( 1e2 / (1e-5 + z), 1e-2, 3e3 );
      gl_FragColor = vec4(gl_FragColor.rgb * gl_FragColor.a, gl_FragColor.a) * weight;
    }
    else if( oitMode == 1 ) { // Revealage Pass
      gl_FragColor = vec4(gl_FragColor.a);
    }

#endif

oit_pars_vertex.glsl

#ifdef USE_OIT

  varying float vPositionZ;

#endif

oit_vertex.glsl

#ifdef USE_OIT

  vPositionZ = mvPosition.z;

#endif

And if a float buffer isn't available, we simply didn't use OIT falling back to standard sorted transparency.