renderdoc crashes with std::bad_alloc when calling CreatePixelShader

[jonasmr]

When I compile shaders with debug information, I get a crash in CreatePixelShader, when launching my engine through renderdoc.

The callstack(the renderdoc part of it)

renderdoc.dll!000007fed6d418cb()    Unknown
renderdoc.dll!000007fed6e0f544()    Unknown
renderdoc.dll!000007fed6e1acdf()    Unknown
renderdoc.dll!000007fed6de0180()    Unknown

heres the module load information:
renderdoc.dll C:\Program Files\Crytek\RenderDoc\renderdoc.dll N/A N/A Cannot find or open the PDB file. 126 0.18.0.0 2/25/2014 4:53 PM 000007FED6D20000-000007FED6FBE000

Let me know if you need more information. I dont Think I can share the shader byte code unfortunately.

[baldurk]

From the callstack it looks like you're hitting an unhandled or buggy case in my shader disassembly code that I haven't seen before. I don't see any obvious way to break it that I've missed in that bit of the code, though.

Without being able to get the shader blob that you pass to CreateShader, the easiest way to track down the problem would be if you could contact me directly at renderdoc@crytek.com. I can arrange getting a build with very verbose logging of the disassembly to you, and then from the last lines in the log I should hopefully be able to track down the bug.

[jonasmr]

Thank you, I've sent you a mail.

[gilenei]

Hi,

Just wanted to let you know I bumped into the same problem:
msvcr100.dll!memcpy() Line 272 Asm
renderdoc.dll!000007fee6e31914()
[Frames below may be incorrect and/or missing, no symbols loaded for renderdoc.dll]
renderdoc.dll!000007fee6eff544()
renderdoc.dll!000007fee6f0acdf()
renderdoc.dll!000007fee6ed0180()
hide_exe.exe!GraphicsDevice11::CreatePixelShader(const void * byteCode, unsigned __int64 byteCodeLength, ID3D11ClassLinkage * classLinkage) Line 482 + 0x5a bytes C++

Of the top of my my head, could be just a memory out of bounds transfer attempt.
(If you could send me a patched exe after you fix the issue it would be wonderful :) )
Thanks,Cheers.

[baldurk]

@gilenei feel free to get in touch via email and I can send you the same verbose logging build & follow up patch - it could be that you're running into a different disassembly problem. Given the complexity of decoding the dxbc bytecode I'm not expecting there to just be a single bug for me to fix :).

[gilenei]

Hi baldurk, I sent you an email from my personal address but I didn't get anything from you yet. Just want to make sure you got it. Just in case: gilenei@yahoo.co.uk
Thanks,Cheers

[baldurk]

Thanks to gilenei I found that his crash was due to issue #5. This could explain other people's crashes also, but I won't mark this bug as a duplicate just yet until I've proven that there isn't a different crash in disassembly.

[mkgski]

Hi Baldurk,

I'm getting a similar crash during CreatePixelShader, though it is in a strrchr looking for forward slashes in a NULL string instead of memcpy like the above callstacks.

msvcr100.dll!strrchr(const char * string=0x0000000000000000, int ch=47) Line 42 C
renderdoc.dll!000007feeb62ddc7() Unknown
renderdoc.dll!000007feeb62ac1b() Unknown
renderdoc.dll!000007feeb5f0180() Unknown
(CreatePixelShader)

I've looked to see if there were any #line or end-of-file symbols and I believe I've removed them all to no avail, though I may have missed some. Is there any possibility of getting a drop of 0.19 to see if my issue is still occurring with your latest fixes?

Thanks

[baldurk]

Checking the code it's a very obvious mistake on my part, I first checked for '/' then for '' without checking the string. Oops :). It's been fixed incidentally with my refactoring to fix the other bug.

I'm hoping to get v0.19 out by the end of this week, and I'm on track for that (not counting any time taken between me passing over a build on friday morning and it getting put up on the website).

[mkgski]

Great to hear, thanks!

[baldurk]

I'm going to close this bug as I think I've fixed the issues seen so far. If anyone experiences any bugs similar to this after 0.19 please file a new bug!