Render Antialiasing to Stencil Buffer

[kkaefer]

When rendering the antialiasing outlines for polygon, we should render them to the 8-bit stencil buffer and then somehow use that mask to affect opacity. This allows us to do alpha-blending of the entire polygon without having a special outline that has different shading/color than the rest of the polgyon area.

[ansis]

If I'm understanding things correctly, the only way to change the value of the stencil buffer is to set stencilOp to invert, increment, decrement, or set a constant (per draw call value). I'm not seeing a way to write (or then use) arbitrary fragment-shader-determined values.

I can't see how using the stencil buffer to do semi-transparent masking is possible.

[ansis]

I'm thinking about this in the context of #77 now. If a draw call draws any fragments that aren't opaque, then it can't be drawn out of order (using the depth buffer). It might make sense to draw anti-aliasing separately for all features so that we can take advantage of depth culling and sorting.

I think polygon anti-aliasing could be nicely drawn using the depth buffer (instead of the stencil buffer). One call which draws the main polygon and writes to the depth buffer. A second call that draws anti-aliasing and avoids the main polygon through depth culling. Most of the fragments in the second call will be depth culled, so theoretically this should perform better than the current approach.

[ansis]

... and I just realized that we need the stencil buffer for concave polygons, so much of what I wrote above doesn't apply.

[kkaefer]

We already cull the antialiasing fragments that lie inside the polygon. This doesn't seem worth investigating now.