Align sublayer depth calculation

[kkaefer]

Followup from #9699: We're using different formulas to compute the depth mode:

JS:

setDepthSublayer(n: number) {
    const farDepth = 1 - ((1 + this.currentLayer) * this.numSublayers + n) * this.depthEpsilon;
    const nearDepth = farDepth - 1 + this.depthRange;
    this.gl.depthRange(nearDepth, farDepth);
}

Native:

gl::DepthMode PaintParameters::depthModeForSublayer(uint8_t n, gl::DepthMode::Mask mask) const {
    float nearDepth = ((1 + currentLayer) * numSublayers + n) * depthEpsilon;
    float farDepth = nearDepth + depthRangeSize;
    return gl::DepthMode { gl::DepthMode::LessEqual, mask, { nearDepth, farDepth } };
}

We should fix this difference. Once it's fixed, we also have to reverse the logic in render_fill_layer.cpp that switches between 0 and 2 for outline sublayers.

/cc @jfirebaugh

[stale]

This issue has been automatically detected as stale because it has not had recent activity and will be archived. Thank you for your contributions.