Camera2D360 (Riemann sphere)

[claudeha]

Is your feature request related to a problem? Please describe.
2D fractals can be reprojected in 360degrees (inside the Riemann sphere). FragM's 3D camera has an equirectangular mode but the 2D camera doesn't. The controls needed for Riemann sphere 360 equirectangular projection would probably not be too hard to add to the Camera2D, so no need for a separate Camera2D360 API.

Describe the solution you'd like
The complex plane can be represented as a sphere. Moebius transformations conformally transform this Riemann sphere, encompassing all of rotation, translation and zooming in one 2x2 matrix of complex numbers (normalized with the condition det M = 1 which reduces the degrees of freedom).

The Moebius stuff would be hidden inside the Camera2D C++ code, activated when the proposed new uniform bool Equirectangular is enabled (or maybe uniform int Projection would be more future-proof?). The C++ would translate to/from regular Center + Zoom (+ Rotation + Stretch + StretchAngle when transformation is enabled) plus an additional uniform vec4 Orientation which would be a unit quaternion to orient the equirectangular projection (so the zoom center does not need to be the center of the equirectangular view).

The idea came from a comment on https://www.youtube.com/watch?v=5_v2VYZRKeg :

What if there was a way to use this projection method but be able to "move" freely in all directions to explore the mandelbrot?? That would be INSANITY! -- realflow100

Describe alternatives you've considered

• separate Camera2D360 API (pro: probably simpler to implement to get started; con: need to modify frags to switch camera)
• try to do it frag-only without C++ code (con: impossible to get sensible mouse interaction)

Additional context
If this would likely be accepted, I can work on a pull request in the coming months (still need to figure out all the maths, then implement it, test existing frags with it disabled and enabled, both single and double precision, etc)

[3Dickulus]

so... basically, render equirectangular, display as 3D... I do not want to have to change all the frag sources, that will put every user's collection in a terrible state of disarray having to adapt any changes they may have made to existing raytracers to a new batch or having to maintain 2 separate collections.

I like the video, pleasantly different compared to ultra boring deep zooms. 👍

1. FragM has equirectangular for making videos like that already (imho no real "need" to do this)
2. a lot of overhead rendering parts that can't be seen, gotta render it because the viewer "might" look in that direction
3. is it really helpful for fractal exploration or just a novelty?

could a 2D 360 camera type be added to the Camera class? so from a user perspective it's very simple, just change Camera2D in their frag to CameraR2D (Riemann 2D) which would use existing equirectangular rendering code.

[claudeha]

render equirectangular, display as 3D

I think for this it makes more sense to display as equirectangular too (like the 3D equirectangular option does).

I do not want to have to change all the frag sources

and

just change Camera2D in their frag

seem to conflict?

I could implement it as separate Camera class type, but adding a new option to the existing 2D camera would avoid having to change any frags.

I would make sure the old Camera2D.frag implementation still works as expected, just without the new feature, so old exported frags in the wild don't break.

All of the bundled frags would automatically gain a new (optional) feature (the default would remain regular flat projection) with changes to the bundled Camera2D.frag (or whichever frag(s) needs the changes).

There is a precedent for this: when I added the rotate+stretch transformation options to the 2D camera.

FragM has equirectangular

Only for 3D afaik, and it's an option in the Camera3D class (no separate Camera3DEquirectangular type).

rendering parts that can't be seen

That's not what I meant. Display as eqiurectangular too (at least for a first implementation).

Is it really helpful for fractal exploration or just a novelty

Can't really say which until I've implemented it to be able to try it out! And implementing in FragM would benefit more people than if I implemented it in a separate toy renderer...

[3Dickulus]

ok

just change Camera2D in their frag

was thinking a flag like some of the other "built-in" things like DepthToAlpha and AutoFocus, thinking more about it makes sense, new camera option vs new camera type

FragM has equirectangular

...a template in the code for adding a "projection" option...

could an existing 3d frag using skybox render a 2D fractal with Riemann projection ? a Box + Riemann projection? then enable equirectangular to render frames for a 360 video.

current 3d box...

#define providesBackground
...
uniform samplerCube skybox; file[cubemap.png]
vec3  backgroundColor(vec3 dir) {
float t = length(from-dir);
dir *= -1.;
    return mix(textureCube(skybox, dir.xzy).rgb, BackgroundColor, 1.0-exp(-pow(Fog,4.0)*t*t));
}

...could the backgroundColor function be use to return a Riemann projection of 2D fractal?

is there a Riemann projection matrix that can be used like the GL matrices proj * view * obj to get hardware to do the work?

I like the video, pleasantly different compared to ultra boring deep zooms. +1

just some random thoughts

[claudeha]

could an existing 3d frag using skybox render a 2D fractal with Riemann projection ?

I think the scaling would be wrong (and non-conformal): as you approach a sphere from the inside, the opposite side reaches a fixed limit size instead of keeping shrinking forever. And worse, the skybox you ref'd is infinitely far away, so you can't even get closer to it to magnify a part...

I looked at the source for 3D equirectangular and it doesn't do anything special with navigation, so mouse controls are a bit weird (e.g. dragging downwards at the edges is the same as dragging down in the middle which moves the image at the edges upwards, when ideally they would drag the image with them pushing the middle upwards).

Here's a sketch of the maths involved with 2D 360 (untested...):

vec2 proj360(vec2 d) // d in [0..1)x[0..1)
{
  float theta = d.x * (pi * 2.0);
  float phi = clamp(2.0 * d.y - 1.0, -1.0, 1.0) * (pi / 2.0);
  vec3 v = vec3(cos(theta) * cos(phi), sin(theta) * cos(phi), sin(phi)); // equirectangular to sphere
  vec2 e = v.xy / (1.0 - v.z); // stereographic projection from Riemann sphere to 2D (infinite) plane
  e = cDiv(cMul(A, e) + B, cMul(C, e) + D); // Moebius transformation, det [A,B;C,D] = 1
  return e; // on whole extended complex plane (including infinity)
}

It might be worth thinking about using dual numbers for automatic differentiation, for scaling DE, in case dFdx() and dFdy() afterwards are not accurate enough or have glitching issues due to non-uniform control flow. But that could be done in frags or not, wouldn't affect the C++ API at all.

The Moebius transformation combines any conformal rotations/translations/scalings, but not non-conformal skew/stretch (not sure how to support that in this projection yet, would definitely be desirable though. Maybe two Moebius transformations would be necessary, bracketing the stretch:

  ...
  vec2 e = v.xy / (1.0 - v.z); // stereographic projection from Riemann sphere to 2D (infinite) plane
  e = cDiv(cMul(A1, e) + B1, cMul(C1, e) + D1);
  e = stretchtransformmatrix2x2 * e; // fixes 0 and infty
  e = cDiv(cMul(A2, e) + B2, cMul(C2, e) + D2);
  return e;
  ...

Then mouse navigation gets really complicated, what to change and how. Each mouse action would be a Moebius transformation that should be combined with the existing state. In the case with no stretch (identity matrix), maybe the two state Moebius transformations could be equalized, making each a square root (see: https://mathr.co.uk/blog/2015-02-06_interpolating_moebius_transformations.html ) of a third Moebius transformation that could be controlled by navigation?

Not written any code yet, still speculating and thinking of the best design...

[3Dickulus]

dragging downwards at the edges is the same as dragging down in the middle which moves the image at the edges upwards, when ideally they would drag the image with them pushing the middle upwards

for 3D eq-projection (in my head) it actually makes sense the way the image responds, you mean the image, at the click point should move the same direction as the mouse? the horizon forms a sine wave, centre moving up, sides moving down, there's 2 points where the horizon doesn't move up or down, what happens when you click there? Same thing happens with linear movement ie:when target and camera move the same so the camera angle doesn't change, T key, translate up, G key translate down and not unexpectedly, Y key left, H key right don't cause changes at the horizon.

the skybox you ref'd is infinitely far away, so you can't even get closer to it to magnify a part...

FOV seems to get extremely close to the cube surface but that's not the desired effect.

[claudeha]

I meant the image should follow the mouse, yes. The vertically fixed points on the horizon will be at +/-90deg from the mouse click, so you can't click there (unless you have two mice - multitouch would be another thing to consider altogether...).

The way the mouse works for 3D equirectangular is fine I suppose for that use case, but for the 2D thing it would really need to be handled differently (I expect; still not had time to write any code or work out the maths).