Compile shaders when a scene is loaded

[endragor]

Operating system or device, Godot version, GPU Model and driver (if graphics related):
Godot af27414, any OS

Issue description:

Currently shaders are compiled lazily - whenever a shader needs to be invoked, glCompileShader is called. This results in noticeable freezes in-game when some object is rendered for the first time. Behaviour that would make more sense is to compile the shader when a scene depending on it is loaded.

[Zylann]

Or more precisely, maybe compile the shader when its resource is loaded, which is actually what would be expected? It allows to load them on loading screen / in background without needing a scene, and if they are embedded in the scene it would still work as expected

[reduz]

You don't know what to compile until you render it. Also even with that, OpenGL driver can do a lazy compilation anyway. The solution for this is shader caching, which will need to be implemented in 3.1

…

On Mon, Dec 4, 2017 at 8:24 AM, Marc ***@***.***> wrote: Or more precisely, maybe compile the shader when its resource is loaded, which is actually what would be expected? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#13954 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AF-Z291pfd4DG8ImBb_4sqyUwzKqsJwEks5s89ZSgaJpZM4Q0bDZ> .

[bruvzg]

Is there any way to cache compiled shaders besides glGetProgramBinary?
It is supported only in OpenGL 4.1+/GLES 3.0+, and Godot targets GL 3.3 on desktop platforms.

[fracteed]

Ah, so this is why I have been getting all the pauses at the start of my game! Always noticed this when bullets, bonuses, explosions are being spawned the first time. Thought it was the particle systems, but it must be the shaders being compiled...makes sense now :)

To avoid this, does the shader have to be seen on an object to compile or can I put them on meshes or particles that are behind other objects out of sight?

[Calinou]

It is supported only in OpenGL 4.1+/GLES 3.0+, and Godot targets GL 3.3 on desktop platforms.

From what I've heard, Godot 3.0 doesn't run on GeForce 8/9 GPUs anyway (which support OpenGL 3.3 at most). I've heard conflicting reports about Radeon HD 6000 GPUs (which support up to OpenGL 4.2, IIRC, which is enough for that feature). As for Intel graphics, you need an Haswell IGP at least (older generations are too slow to be useful for modern gaming anyway).

So, we may not need to bother about keeping OpenGL 3.3 compatibility, unless you want to keep compatibility with software fallbacks (see Mesa matrix), which are extremely slow anyway – their only use is for vendor-independent OpenGL testing.

[LinuxUserGD]

I've noticed that loading animations in the ResourceInteractiveLoader (background loading) freeze if the scene contains 3d objects with lots of materials. Currently I spawn all objects at loading time and call remove_child to keep them in memory.

[eon-s]

Probably related to #17263 (but particles even freeze the game with no materials!).

I hope this gets in 3.1 because it makes mobile games unplayable.

[ProbDenis]

Yeah, it's really a problem. I tried to display all important materials when the game starts, and that seems to have improved it a little - but the particles still cause these freezes (maybe because the ParticlesMaterial has to be compiled too?) It would be insane to run every existing particle effect at the start of the game.

[LinuxUserGD]

On html5 (3d game) it takes more than 10 seconds to pre-render 3 cars offscreen in a car selection menu.

[fracteed]

@ProbDenis I have also tried these workaround, but still get the pauses on the first time particle systems are instanced. I was also guessing that it was that the particle system material that needs to be compiled. This is the main issue I am having with my game (along with lights turning themselves on/off at random). I am pinning my hopes on Vulkan to resolve these issues!

[endragor]

You can make particle materials pre-compile the same way - just add hidden particles with materials that can be used in the scene during your loading screen.

[eon-s]

If you have, lets say, an endless runner with a few different little effects, the loading screen will be unresponsive for half minute.

There should be a real solution for this (I mean, I have never seen this issue on the rest of the engines, with few materials), I do not know if that can be some kind of threaded process or compilation queued in frames or what.

[Zylann]

Half a minute for a few materials sounds really long... maybe the generated shaders themselves are quite big, or the optimizer is having a hard time crunching if branches? (if there is an optimizer though)

[endragor]

@eon-s Well I make a mobile game that has lots of particles and the loading screen between missions doesn't hang for half a minute, it takes few seconds on an average device. Did you base your statement on an actual experience or was it a guess?

[eon-s]

The super simple coin effect on the platformer demo takes from 3-5 seconds to load on mobile, for example.

[LinuxUserGD]

@eon-s I think it's a driver problem in this case. 20 materials should be loaded in ~ 5 seconds (even on mobile), html5 takes much longer.

[Zireael07]

@LinuxUserGD: It's not just loading the materials, it's also compiling the shaders so it takes a lot of time.

[endragor]

Compiling shaders doesn't take that much time. The game I mentioned compiles 100-200 different shaders during loading screen and it takes less than 5 seconds on mobile. So either there is something with that shader specifically, or with the specific GPU that you test on, and it needs to be investigated as a separate issue. Even if pre-compilation of shaders is implemented, it will not solve the problem you describe.

[endragor]

Ah, but the game uses a fork of Godot with PR #14902 merged into it, which significantly reduces compilation time of shaders (see "Issue 2" in the description of the PR). It wasn't merged into master though and I don't know why.

[eon-s]

That was not merged because is a hack for Adreno and do not solve all the issues with that chip.

Could be nice to be able to set those things as setting for android and wasm, if these huge and normally useless arrays are the problem, maybe a pr only for that option (limit on some arrays) can solve a lot of issues.

[endragor]

Well it's a "hack" just as much as other hacks existing Godot codebase and really any other large codebase that solves real-world problems. There are no side effects created by that PR that I'm aware of and it solves plenty of issues, while not complicating the architecture. So I think the benefits outweigh the vague "it's a hack" argument.

[fracteed]

@endragor I had previously tried this with particles, but still seem to get some pauses the first time they are truly used in the game. When say "add hidden particles" at the start of the game do you mean that they are emitted offscreen or behind geometry or maybe onscreen but with .hide() on? I am wondering if culling is why it is not working for me.

When you say that you have 200 materials how are you displaying these. I have tried having all my materials on cubes in a separate scene and then instancing this for a frame at the start of the game, which does seem to help but not totally alleviate the problem. Maybe there is a way with the profiler or monitor to know when a shader is being compiled?

[endragor]

We have a ShaderCacheNode that is added to the level scene. During loading screen it's supplied with PackedScenes that could be instantiated in the level, extracts materials from them, then creates either a Skeleton or Particles or MeshInstance (quad), depending on where the material is used in the scene. The created node is hidden (node.hide()) and is added as a child to ShaderCacheNode. When addChild is called, the shader is compiled. ShaderCacheNode exists for the whole duration of the level to ensure the shaders are always there and are compiled.

[Zireael07]

@fracteed: No way to tell when a shader is being compiled. I have a PR in progress that separates rendering from 'idle' for profiling, which will hopefully let us tell the other stuff better: #18357

[fracteed]

@endragor thanks that is very useful info! It is similar to what I had been trying but I was just doing it manually and instancing in ready function and then freeing. I have tried your way of parenting to a hidden dummy node and it does seem a bit better. Still getting some random pauses at the start, but only sometimes, which is frustrating.

Just wanted some more clarification on what will trigger a shader to be compiled:

• does the object with the shader have to be in the view frustrum, or will culling it cause it to not compile the shader? Can it be behind an object or will this also stop the shader compiling?
• by the sounds of it, a hidden object's shader will compile, but why does the object have to stay in the scene? Will it be decompiled if it is freed?
• with particle systems, there is the process material and the actual spatial shader on the particle. Is the process material also compiled on the gpu? Does the particle system need to start emitting (even though hidden ) to be compiled, even if the spatial shader for the particles are already on hidden quads?
  cheers!

@Zireael07 this sounds like a very useful feature, hopefully it will be committed!

[snoopdouglas]

I've always thought this was a bit out of its remit, but Steam caches shader binaries for even faster loading.

[Zireael07]

@akien-mga: I think some games do that (e.g. FUEL). Any way to imitate such a loading screen in Godot?

[snoopdouglas]

@Zireael07 I guess you'd want to follow the video's instructions, but make each material visible in turn as part of your loading screen.

[Zireael07]

@snoopdouglas: Thing is, FUEL's loading screen is just a bar on some static 2d background, the materials aren't visible to the end user.

[snoopdouglas]

@Zireael07 Did you watch the video? They accomplished it by adding less-than-1x1 pixel meshes in front of the camera. Each one had a different material. They aren't visible so far as the player's concerned.

More to the point, though, is that this is a workaround for a Godot-specific problem. In other engines, I have more fine-grained control over when the shaders are compiled, so I don't have to resort to janky solutions like that.

[eon-s]

I have seen it on some Unity mobile games, you can actually see the dot where materials are placed on screen if your device is sufficiently bad...

If any of these solutions mentioned in the last 3 years is generic enough, maybe it could be considered for adding to the engine? (export options, perhaps?)

[remram44]

I was dealing with this today, because I exported a game to HTML5, where the pauses are most noticeable. Having this pre-loading is fine, however I'm not sure I understand how it works. How can I prevent the compiled shared to be thrown away and recompiled? Do I need to keep the reference to load("res://xxx.tscn") (PackedScene) alive? Is the cache shared between all scenes loaded from the same file, or do I need to pass that reference around? Is just the material sufficient? Is showing those scenes for 1 frame enough or do you need to wait until some frames actually get rendered on screen (which on web can take long)?

[snoopdouglas]

To be clear: even if Unity does share this issue, that is nothing more than a common design complaint. In plain old OpenGL, shaders are compiled when the programmer says so.

Keeping in mind that I'm not a generalist on Godot's internals, I would interested to know from one of the developers why the actual call to glCompileShader can't be moved into the resource loader. Presumably it's something to do with (for example) SpatialMaterials changing their parameters at runtime, but yeah, a slightly lower-level explanation would be interesting to read.

[clayjohn]

@snoopdouglas it's because you are not writing OpenGL shaders directly. You are writing Godot shaders. Godot abstracts away all the nasty bits of shader writing and presents something much easier to the user.

The shader you write does not map 1-1 with an OpenGl shader either. When you create a shader multiple versions are created internally (for depth prepass, shadow pass and drawing pass).

Further, shaders are generated based on parameters in the Godot shader (for example turning on and off features in SpatialMaterial, or whether you use a feature in a regular shader). So a given Godot shader can map onto completely different OpenGL shaders at runtime and at load time.

If shaders were compiled at load time a) users would lose control over loading screens, b) the hiccups would resurface whenever a shader or material is changed, and for all shaders created at run time and c) you would end up needlessly compiling shaders that are never run.

The easiest solution is to just compile a shader when you know it is going to be used (i.e. directly before it is used). The best solution is pre-compiling and then caching shaders which is why we are moving to that in 4.0

I hope that explanation is helpful! I know the compiling hiccup is an annoying issue and the workaround is not fun, but it's the best we can do right now.

[snoopdouglas]

Thanks @clayjohn - this makes the whole thing a ton clearer.

The best solution is pre-compiling and then caching shaders which is why we are moving to that in 4.0

Perfect.

[Zireael07]

AFAIK that precompiling and caching is a part of Vulkan, right? So it one used GLES2, it isn't a thing, right?

[starry-abyss]

I looked at engine code at some point, and shader compilation works the same in GLES 2 and GLES 3 - at first use. I don't have this issue (stuttering) in GLES 2. Maybe this is because shaders are simpler/smaller in this renderer?

[ron0studios]

I solved the issue myself with a temporary loading screen. Has shader caching and the lazy compilation been fixed yet?

[Calinou]

Has shader caching and the lazy compilation been fixed yet?

The Vulkan renderer in the master branch should no longer exhibit pauses during gameplay, although shader caching hasn't been implemented yet.

[remram44]

The worse pauses are experienced on webgl though, and I assume the Vulkan work won't help there.

[rogerdv]

Has shader caching and the lazy compilation been fixed yet?

The Vulkan renderer in the master branch should no longer exhibit pauses during gameplay, although shader caching hasn't been implemented yet.

I tried yesterday, with master updated a week ago. I still get a pause when an object in the scene becomes visible.

[KoBeWi]

Should be resolved by #49050

[TheConceptBoy]

Like what did they expect games to do? Just stutter like crazy or just have crazy load times that completely freeze with no feedback to compile all possible shaders?

I would expect that many games with loading screens actually freeze rendering while compiling shaders incrementally:

• Show loading screen, 0 %
• Compile shader 1
• Increment loading bar by 1/shader_count
• Compile shader 2
• Repeat

If only we could load and compile materials in a separate thread. Fallout 4 has that whole 3D model viewer in the loading screen that seems to be unaffected by loading and compiling materials on the fly.

I find that when you queue_free an object that uses a material, that material is also unloaded, meaning that next time I instance the same object, all the materials it uses have to be compiled again

[TheDuriel]

If only we could load and compile materials in a separate thread. Fallout 4 has that whole 3D model viewer in the loading screen that seems to be unaffected by loading and compiling materials on the fly.

The first of these models are loaded in the title screen. Additionally, they are single models, and you can see them pop in as they are loaded when it needs a new one.

Like what did they expect games to do? Just stutter like crazy or just have crazy load times that completely freeze with no feedback to compile all possible shaders?

This is actually the industry standard right now, unless you're call of duty and can download compiled shaders for every possible hardware config. You will find this problem in many prominent titles, as a byproduct of just how expensive shader compilation has gotten over the decades. As the standards were only ever extended, never rethought.

ANYWAYS. It's a dead conversation. Wait until 4.0, see what can be improved at that point.

[Calinou]

For reference, this is addressed in 3.x by #53411 (and in master by #49050).

If you're interested in further reading on the subject (not Godot-specific):

• https://therealmjp.github.io/posts/shader-permutations-part1/
• https://therealmjp.github.io/posts/shader-permutations-part2/

To give a point of comparison, this issue has been increasingly plaguing Unreal Engine 4/5 releases on PC too. Recent titles like Shadow Warrior 3 or Ghostwire Tokyo were released with shader compilation stutters the first time a new shader appears on screen, especially for particle effects.

Consoles manage to sidestep this problem by having a unique hardware configuration, with precompiled shaders shipped with the game data.