Debug builds on windows result in "thread 'main' has overflowed its stack"

[seratonik]

• Version of vulkano: Git master branch at 965c9f1
• OS: Windows 10 Pro 64bit Build 19042.1052
• GPU (the selected PhysicalDevice): NVIDIA GeForce RTX 3090
• GPU Driver: 471.11

Issue

Something that changed after commit 3850a92 seems to be causing a stack overflow error when I attempt to run my game that uses Vulkano, but it only happens in Windows and only when compiling in debug mode. Compiling on OSX or in Release mode on Windows fixes the issue.

I don't have a small reproducible example at the moment, I just wanted to report it when I saw it. I also had issues with this latest version crashing rust-analyzer's macro system in VS Code in both Windows/OSX when using vulkano_shaders::shader!

[Eliah-Lakhin]

@seratonik The commit you referred to is just a readme update. If the issue didn't exist before, it looks more like some "caching" issue or an issue with dependencies or an environment configuration at a first glance. Also it unlikely that any recent changes in vulkano-shaders could lead to crashes. Can you try to find an exact commit from which your project works? Or at least at which Vulkano version.

[funmaker]

I have the same issue in my project. I think the cause is surprisingly large size of GraphicsPipelineBuilder struct. In 0.23 it was 352 bytes, 1072 in 0.24, and 84232 in 0.25. In debug builds, each invocation of builder methods adds another 82kb to the stack. This can quickly overflow even in regular usage.

Reproduction:

use vulkano::pipeline::GraphicsPipeline;

fn main() {
	GraphicsPipeline::start()
	                 .cull_mode_back()
	                 .cull_mode_back()
	                 .cull_mode_back()
	                 .cull_mode_back()
	                 .cull_mode_back()
	                 .cull_mode_back()
	                 .cull_mode_back()
	                 .cull_mode_back()
	                 .cull_mode_back()
	                 .cull_mode_back()
	                 .cull_mode_back()
	                 .cull_mode_back()
	                 .cull_mode_back();
}

[Chris--B]

I tracked down some sizes of things and I think the issue is a SmallVec optimization in GraphicsEntryPoint (edit: Also applies to ComputeEntryPoint).

GraphicsEntryPoint looks like this:

pub struct GraphicsEntryPoint<'a> {
    module: &'a ShaderModule,
    name: &'a CStr,

    descriptor_set_layout_descs: SmallVec<[DescriptorSetDesc; 16]>, // <---- !!!
    push_constant_range: Option<PipelineLayoutPcRange>,
    spec_constants: &'static [SpecializationMapEntry],
    input: ShaderInterface,
    output: ShaderInterface,
    ty: GraphicsShaderType,
}

and is 16,776 bytes. But DescriptorSetDesc is 1,040 bytes, so this inlining optimizations explodes the size of the struct. I believe the rest of the fields add up to 136 bytes, including padding.

I found this commit: 1771714
It refactored some types and moved this:

/// Object that describes the layout of the descriptors and push constants of a pipeline.
#[derive(Debug, Clone)]
pub struct PipelineLayoutDesc {
    descriptor_sets: Vec<Vec<Option<DescriptorDesc>>>,
    push_constants: Vec<PipelineLayoutDescPcRange>,
}

into the struct we have now, replacing the pure-heap Vecs with the SmallVecs.

I found some discussion about linting for type sizes in clippy
rust-lang/rust-clippy#4831 (comment)
rust-lang/rust-clippy#6560

An aside:
If you take the previous repo case and add approximately 10x cull_mode_back() calls, the stack overflow reproduces in macOS. macOS and Linux generally default to a stack size of ~8 MB, vs Windows' ~1MB, and I think that explains why this happens on Windows "only".

[PeterButler41]

This simple program produces thread 'main' has overflowed its stack:
If attach did not work...
`//use std::intrinsics::wrapping_add;

type MyRand = [u32; 10];

fn mix(se: &mut MyRand) {
se[3] = (se[3].wrapping_add(se[0]))^se[2].rotate_left(1);
se[4] = (se[4].wrapping_add(se[1]))^se[3].rotate_left(1);
se[5] = (se[5].wrapping_add(se[2]))^se[4].rotate_left(1);
se[6] = (se[6].wrapping_add(se[3]))^se[5].rotate_left(1);
se[7] = (se[7].wrapping_add(se[4]))^se[6].rotate_left(1);
se[8] = (se[8].wrapping_add(se[5]))^se[7].rotate_left(1);
se[9] = (se[9].wrapping_add(se[6]))^se[8].rotate_left(1);
se[0] = (se[0].wrapping_add(se[7]))^se[9].rotate_left(1);
se[1] = (se[1].wrapping_add(se[8]))^se[0].rotate_left(1);
se[2] = (se[2].wrapping_add(se[9]))^se[1].rotate_left(1);
mix(se);
}

fn seed(se: &mut MyRand, val: u32) {
se[0] = se[0].wrapping_add(val.rotate_left( 1));
se[1] = se[1].wrapping_add(val.rotate_left(12));
se[2] = se[2].wrapping_add(val.rotate_left(21));
show(se);
}

fn random(se: &mut MyRand) -> u32 {
mix(se);
let i:usize = (se[9]&7) as usize;
let mut j0 = se[i+0]; j0 = j0.rotate_left(01);
let mut j1 = se[i+1]; j1 = j1.rotate_left(12);
let mut j2 = se[i+2]; j2 = j2.rotate_left(21);
j0.wrapping_add(j1)^j2 //return
}
fn show(se: &MyRand) {
let mut i = 0;
while i < 10 {
println!("{:2} 0x{:08X}", i, se[i]);
i = i + 1;
}
println!("------");
}
fn main() {
let p = &mut [0,0,0,0,0, 0,0,0,0,0];
seed(p, 0x4321);
mix(p);
show(p);
let mut res = random(p);
println!("{:08X}",res);
res = random(p);
println!("{:08X}",res);

}