Concepts I'm developing into a custom implementation

[FishOfTheNorthStar]

Hi all, been a while, just wanted to say hello and outline some of what I've been working on here. Maybe some of it will be worth a look. It's a continuation / hyper-customization of my PR1 branch.

It has some external dependencies, first off - AsyncLiteDB, for storing relevant data. Also two main groups of additional classes I'm calling Blobs and MaryNodes so far. Mary is a bit of a misnomer, technically they'd be M-Ary nodes, like a spatial database, or a geo-cache maybe. Blobs are what you'd expect: image data in blob format. They're compressed and decompressed when stored, and the three main images used to generate the final terrain textures takes about 20mb for a 16 region terrain. Those images are a bit odd though, I generate three kinds of noise, the first is very broad, gradual shifts and that's the overall world elevation at that point, then another smaller grain noise, full spectrum, and that represents smaller adjustments to the world elevation. Then the third map is a biome map, and is very broad and smooth, also full spectrum, and which biome a position is within is determined by taking the floor of texture value times max_biomes. Then each biome can itself specify it's own auto-texturing base/over materials, as well as a Curve2D texture that represents how the second noise texture above is blended with the first. This is so things like desert steppes and such can have a stair-cased look, or mountains can have areas of gradual slope versus sharper peaks, etc.

The noise that's generated within these textures utilizes a sort of FastNoiseLite markup language I've implemented that's a bit like HTML for noise-gen, and allows various factors to be adjusted in a way that can be hard coded as constant. Currently it happens CPU side and it's a bit slow, taking about 15 seconds to generate 16 regions at full resolution.

Notably, blobs don't need to be at full resolution, some of them do better at some fraction of that. All of them are usable at LOD increments, and they render in the same increments like it'll do a render at 1/8th resolution, it can update the terrain from that, then next pass it'll do it at 1/4 resolution, etc.

Those are the only three images stored to create the terrain. Those are then fed into a series of shaders that off-screen render derived textures, which ultimately become the textures Terrain3D uses.

The MaryNodes are for world-stuff like trees, shrubs, smaller plants, and eventually things like buildings and notable stuff. They are used in various stages of the Blob rendering process to generate things like an elevation adjustment layer, and an explicit splat-map override layer

I've customized my control map data structure a fair bit, it's substantially different now. Notably blend resolution has been dropped from 8 bit to 5, which worked out fine for my purposes. Also I have 3 bits dedicated to how much rain can affect the terrain at that point, so areas under trees can be a bit dryer than fully exposed, and interior ground locations can have rain completely omitted.

Transferring 32b floats from the rendering phase I described back into the terrain engine has proven to be unfortunately rather clunky requiring I send the data as RGBA8 and then reconstruct a 32b float from that per pixel cpu side, which is unfortunate because the rest occurs GPU side considerably faster. But SubViewports flatout refuse to provide 32b output at current. For this reason I'm considering other control map formats like maybe RgF, which the SubViewport can provide at 16b per channel, so conceivably I could generate it entirely GPU side. It would require portioning certain control map parameters into one channel or the other though.

One phase of the blob rendering generates a nice Sobel normal map of each region at control-map resolution. This is then used by other phases of the generation. I'll be doing some testing soon to see if loading this generated normal map into the shader and fore-going all other GPU side normal calculation would improve performance, which I expect it would, but the question is would the addition RAM used and texture lookups involved negate those gains. Testing will tell.

Anyways just wanted to say hi and give a little update. None of the above is reflected in my PR1 branch btw that's still how it was when I last mentioned it here. I'll try to push newer changes to it soon for people to try out, with my custom classes linked in somehow, perhaps as a separate project.

[TokisanGames]

Hi, thanks for the update. These sound like interesting projects. Maybe we can mine some of them for inclusion in the upstream. We recently enabled shader includes. I'll be merging a restructuring of the monolithic storage into separate region files tomorrow.

[FishOfTheNorthStar]

Hey that sounds promising, are you thinking about expanding the 16 region limit some? I've been wondering how feasible that'd be, I find myself pushing against it here and wishing 64 or even 256 is possible.

Oh another thing I've tried here I forgot to mention, one of those generated Blob textures is another elevation map, center balanced so 0 elevation is 0.5 float value, and 1.0 is raise X units, 0.0 is lower X units. With an alpha channel to control how much this map gets blended with the actual elevation. This has proven to be extremely useful, maybe you'd find value in it. I've used it for a couple things so far: first, scenery objects that affect the ground in some way, like spacecraft wreckage sites that leave a smoldering crater, or trees with a rootball that somewhat protrudes out of the ground. Each of those examples is implemented as a grayscale image mapped to a TextureRect in 2D, collectively rendered into a 1024x1024 region texture, then blended with actual elevation in another Blob texture render stage, then that's sent to Terrain3D as it's height source. Then I do that again, but generate another offset texture based on the edge to edge mixed splat texture height at a given region map texel, so the splat height affects the actual height, like dynamic texture based tesselation would except the collider remains accurate.

Both of these usage scenarios have really been game changes for me so I wanted to pass that along.

In the future I plan on trying out some erosion simulation with the same basic plan, and the tree root example is actually just a placeholder for building foundations, which will utilize the exact same system.

Rendering these offscreen Blob intermediate textures has proven to be fast enough, perhaps faster than loading them from disk... My current process probably has at least 12 of these intermediate steps involved and aside from per-pixel converting the Control map into 32bit float from 4x8bit (which is crazy slow), the rest happens for 16 regions in about 0.5 seconds here on my machine, which isn't particularly cutting edge. And it's updating the terrain data 3 times in there too so I bet if I really tried to see how fast I could get it, it'd be half that.

btw when I mentioned how rendering the initial noise maps takes 15 seconds, that's a one time expense. Those get saved and cached and after that the terrain loads up it's 16 regions, with a collider in under a second.

I've cooked up a pretty neat imposter/billboard system too. It easily handles thousands of trees, unlimited grass/shrubs etc. It's pretty tough to even tell it's not actual 3D, and I have them turn into actual 3D mesh objects once you get close enough anyways. I might try to package that up as its own thing, maybe we can find a standardized way to integrate it with Terrain3D (the proper one, not just my custom one). Really, it's already integrated in that it's basically agnostic and doesn't care what is providing the ground elevation at a given point or how it does it, just needs a proper delegate registered. But it might help showcase your project some, add a little greenery to the demos etc. Terrains look way cooler with lots of plants.

[TokisanGames]

We've always had a 16 x 16 = 256 region limit, though it couldn't be saved due to an engine bug. We've now worked around that. We're planning on a 2048 (45x45) region limit, as that is the maximum texture array size for most GPUs. Most people don't have enough vram for more than 2048 currently.

The other ideas sound interesting, especially the billboard system. I'd like to take a look.