Debug info: Fixup value out of range

[kinke]

Environment: Win64, VS 2015, LDC merge-2.067 (+ my VS15 patches + kinke/druntime@1add4f0), LLVM master (5de99601367aaf09ba1d17cd51478bde5e1b74fe).

All druntime debug unittests compile fine and all Phobos ones too, except for std.string and only when using the -g switch to generate debug infos. This snippet:

import std.conv, std.string, std.typetuple;

void invoke(alias dg)() { dg(); }

void main()
{
    invoke!({
    foreach (S; TypeTuple!( char[], const( char)[], immutable( char)[],
                           wchar[], const(wchar)[], immutable(wchar)[],
                           dchar[], const(dchar)[], immutable(dchar)[]))
    {
        foreach (T; TypeTuple!( char[], const( char)[], immutable( char)[],
                               wchar[], const(wchar)[], immutable(wchar)[],
                               dchar[], const(dchar)[], immutable(dchar)[]))
        (){
            foreach(R; TypeTuple!(dchar[dchar], const dchar[dchar],
                        immutable dchar[dchar]))
            {
                R tt = ['h' : 'q', 'l' : '5'];
                assert(translate(to!S("hello world"), tt, to!T("r"))
                    == to!S("qe55o wo5d"));
                assert(translate(to!S("hello world"), tt, to!T("helo"))
                    == to!S(" wrd"));
                assert(translate(to!S("hello world"), tt, to!T("q5"))
                    == to!S("qe55o wor5d"));
            }
        }();
    }
    });
}

yields the following when using the -g switch, and otherwise compiles fine:

Assertion failed: isIntN(Size * 8 + 1, Value) && "Value does not fit in the Fixup field", file C:\LDC\llvm\lib\Target\X86\MCTargetDesc\X86AsmBackend.cpp, line 120
    ldc2.exe!`anonymous namespace'::X86AsmBackend::applyFixup(const llvm::MCFixup & Fixup, char * Data, unsigned int DataSize, unsigned __int64 Value, bool IsPCRel) Line 119   C++
    ldc2.exe!llvm::MCAssembler::Finish() Line 962   C++
    ldc2.exe!llvm::MCObjectStreamer::FinishImpl() Line 433  C++
    ldc2.exe!llvm::MCWinCOFFStreamer::FinishImpl() Line 268 C++
    ldc2.exe!`anonymous namespace'::X86WinCOFFStreamer::FinishImpl() Line 48    C++
    ldc2.exe!llvm::MCStreamer::Finish() Line 583    C++
    ldc2.exe!llvm::AsmPrinter::doFinalization(llvm::Module & M) Line 1154   C++
    ldc2.exe!llvm::FPPassManager::doFinalization(llvm::Module & M) Line 1575    C++
    ldc2.exe!`anonymous namespace'::MPPassManager::runOnModule(llvm::Module & M) Line 1632  C++
    ldc2.exe!llvm::legacy::PassManagerImpl::run(llvm::Module & M) Line 1722 C++
    ldc2.exe!llvm::legacy::PassManager::run(llvm::Module & M) Line 1756 C++
    ldc2.exe!codegenModule(llvm::TargetMachine & Target, llvm::Module & m, llvm::raw_fd_ostream & out, llvm::TargetMachine::CodeGenFileType fileType) Line 126  C++
    ldc2.exe!writeModule(llvm::Module * m, std::basic_string<char,std::char_traits<char>,std::allocator<char> > filename) Line 527  C++
    ldc2.exe!main(int argc, char * * argv) Line 1361    C++

We have a llvm::MCFixup of kind FK_Data_2 (16 bits), but its value is bigger (78893 in my case).

[kinke]

When excluding std.string from the Phobos test runner, 91% of the debug druntime+Phobos unittests pass, the following 26 fail:

std.array-debug
std.bigint-debug
std.csv-debug
std.datetime-debug
std.encoding-debug
std.math-debug
std.parallelism-debug
std.path-debug
std.process-debug
std.socket-debug
std.stream-debug
std.traits-debug
std.uni-debug
std.uri-debug
std.zip-debug
std.zlib-debug
std.algorithm.sorting-debug
std.digest.crc-debug
std.digest.md-debug
std.digest.ripemd-debug
std.digest.sha-debug
std.net.isemail-debug
std.regex.internal.parser-debug
std.regex.internal.tests-debug
std.regex-debug
std.internal.math.gammafunction-debug

[redstar]

This is an LVVM bug. You should create a .ll from the D code and run bugpoint on it. The reduced file can then be attached to an LLVM bug report.

[kinke]

Hmm, not sure at which point this happens. ldc2 -g -c crashes, but it's able to produce the corresponding .ll, .bc and .s files. bugpoint -run-llc tells me the .bc is fine, and opt is happily working too. bugpoint seems to be pretty limited without gcc...

[dnadlinger]

Are you running opt with the same set of passes? (Try -std-compile-opts for a start, and passing -debug-pass=Arguments to LDC might be useful. Edit: Hmm, I see, this occurs in debug mode).

You can actually use bugpoint to link an executable with whatever options you need, but it's been a while since I last did that. It won't be relevant here anyway.

[kinke]

I've used '-O3', now checking in combination with -std-link-opts (there's no -std-compile-opts) [edit: that works fine]. Thx for the quick response.

[kinke]

Could this be related to #442? The mentioned LLVM/clang bug seems to be..

[redstar]

Yes, seems to be the same bug.

[kinke]

IR in case someone like @majnemer would be interested: link

[majnemer]

@kinke File is busted for me, I get:
fixup.ll:1484:1: error: expected instruction opcode

[kinke]

Thanks for looking into this too! Line 1484 would still be part of the huge display name extending from the previous line all the way down to line 3963!
Not sure why the .ll file appears to be corrupt. I'll upload a zipped version incl. the bitcode file if that's of any use... here you go.

[majnemer]

@kinke Thanks!

[kinke]

@majnemer With an updated LDC and LLVM (git dbe4385), I now hit another assertion in another place:

Assertion failed: (isUIntN(8 * Size, Value) || isIntN(8 * Size, Value)) && "Invalid size", file C:\LDC\llvm\lib\MC\MCStreamer.cpp, line 82
#5 0x000007f89044116f _wassert (C:\Windows\SYSTEM32\ucrtbase.DLL+0x6116f)
#6 0x000007f7dd15b56b llvm::MCStreamer::EmitIntValue(unsigned __int64,unsigned int) c:\ldc\llvm\lib\mc\mcstreamer.cpp:84:0
#7 0x000007f7dd767c4c llvm::CodeViewDebug::emitTypeInformation(void) c:\ldc\llvm\lib\codegen\asmprinter\codeviewdebug.cpp:278:0
#8 0x000007f7dd768349 llvm::CodeViewDebug::endModule(void) c:\ldc\llvm\lib\codegen\asmprinter\codeviewdebug.cpp:206:0
#9 0x000007f7dd7360cd llvm::AsmPrinter::doFinalization(class llvm::Module &) c:\ldc\llvm\lib\codegen\asmprinter\asmprinter.cpp:1124:0
#10 0x000007f7dcea732a llvm::FPPassManager::doFinalization(class llvm::Module &) c:\ldc\llvm\lib\ir\legacypassmanager.cpp:1589:0
#11 0x000007f7dceaf919 llvm::FPPassManager::runOnModule(class llvm::Module &) c:\ldc\llvm\lib\ir\legacypassmanager.cpp:1645:0
#12 0x000007f7dceaebf6 llvm::legacy::PassManagerImpl::run(class llvm::Module &) c:\ldc\llvm\lib\ir\legacypassmanager.cpp:1731:0
#13 0x000007f7dc9f357f llvm::VectorType::classof(class llvm::Type const *) c:\ldc\ldc\driver\toobj.cpp:96:0
#14 0x000007f7dc9f7745 writeModule(class llvm::Module *,class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> >) c:\ldc\ldc\driver\toobj.cpp:443:0
#15 0x000007f7dc9e50a6 ldc::CodeGenerator::writeAndFreeLLModule(char const *) c:\ldc\ldc\driver\codegenerator.cpp:120:0
#16 0x000007f7dc9e4064 ldc::CodeGenerator::finishLLModule(class Module *) c:\ldc\ldc\driver\codegenerator.cpp:96:0
#17 0x000007f7dc9e3f04 ldc::CodeGenerator::emit(class Module *) c:\ldc\ldc\driver\codegenerator.cpp:158:0
#18 0x000007f7dca0ad0d main c:\ldc\ldc\driver\main.cpp:1284:0

The interesting location is CodeViewDebug.cpp:277, which limits the length of the display name to a little less than 64K. In our case, it's about 80K...

[rainers]

BTW: DMD (the reference D compiler) trims CodeView symbol names to 0xffd8, as the linker bails out with larger names anyway: https://github.com/D-Programming-Language/dmd/blob/master/src/backend/cv8.c#L45

[majnemer]

For the clang/c++ side of things, this is taken care of by an interesting trick: all symbols > 4k get their linkage name md5 hashed. The has is then substituted for the linkage name.

I do not like the DMD approach to this because it silently drops the end of the linkage name. Instead, I'd rather have us skip functions with names that are too large instead of pretending to support the behavior.

Is there a reason why using a scheme akin to the MSVC one would be undesirable for LDC?

[rainers]

For the clang/c++ side of things, this is taken care of by an interesting trick: all symbols > 4k get their linkage name md5 hashed. The has is then substituted for the linkage name.

DMD does similar for the ancient OMF object files: if the (compressed) symbol does not fit into 128(!) characters, it is cut off around 100 and the MD5 hash is appended. I guess we should do something similar for symbols in COFF files.

Is there a reason why using a scheme akin to the MSVC one would be undesirable for LDC?

You mean the compression of symbols with backward references? This should be possible, but the C++ version is rather limited IIRC. OMF symbol compression uses LZW or similar, but generates undecodable symbols in case the name contains non-ASCII characters (D allows any "universal alphas").

The biggest downside to changing the mangling is probably that it has to creep into a number of build tools supporting only the current one.

[kinke]

I'm not closing this yet as I've hit this old assertion just now again with bleeding edge LLVM, for the std.algorithm.setops unittests with debug infos.

[majnemer]

@kinke Could you post the failing IR?

[kinke]

It's 110 MB last time I checked. ;)

[majnemer]

The size isn't a problem for me, I'd still be more than happy to take a look.

[kinke]

But compressed it's only 1.4 MB.
This compiles fine with git dbe4385 from about one week ago. Oh and I got this for today's git a5b740a.

[majnemer]

Figured it out.
Your IR has a function called _D3std9algorithm6setops342__T16cartesianProductTS3std5range57__T8SequenceVAyaa1_6eTS3std8typecons12__T5TupleTiZ5TupleZ8SequenceTS3std5range57__T8SequenceVAyaa1_6eTS3std8typecons12__T5TupleTiZ5TupleZ8S...
The size of the function's name is 139368 bytes and it appears not to have a DISubprogram associated with it.
Because it has no DISubprogram, we use the linkage name for the display name.
This makes the S_GPROC32_ID's record size greater than the maximum of 0xFFFF.

[majnemer]

I ended up fixing this LLVM r263378. We will perform the truncation regardless of where we got the name, you can probably remove the truncation logic on your side.

[kinke]

The size of the function's name is 139368 bytes

😜

it appears not to have a DISubprogram associated with it

Hmm, has a bad smell to it...

I ended up fixing this LLVM r263378. We will perform the truncation regardless of where we got the name

Makes sense imo. Thanks a lot!

[kinke]

@majnemer I've just used current LLVM head. Your fix works for the latest issue, but I still have to truncate the display name of DISubprograms (the original issue). Is that intended?

[kinke]

Well the LLVM truncation fix allows to compile the modules, but then the MS linker complains:

setops-unittest-debug.obj : fatal error LNK1103: debugging information corrupt; recompile module
Error: `"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\amd64\link.exe"` failed with status: 1103

[rainers]

@majnemer One of the truncations uses 0xfff6 as the upper limit (lib/CodeGen/AsmPrinter/CodeViewDebug.cpp, line 712). This seems too large. @kinke, you might want to try reducing it to 0xffd8, too.

[kinke]

Thx Rainer, my machine is already building...

[kinke]

Okay so using 0xffd8 in both locations gets rid of the linker complaints (0xffd9 at the other site was already too high). DISubprogram display names still need to be truncated by us as the original one is used here.

[majnemer]

You can't use 0xffd9 for the variable side of things? Also, I don't understand the bit about DISubprogram. Does it crash in LLVM, what is the failure moe?

[kinke]

You can't use 0xffd9 for the variable side of things?

Yep, otherwise the MS linker would complain for the 'setops' source (x64 at least, not sure about x86).

About DISubprogram: yes I hit this LLVM assertion again when not truncating it on our side.

[majnemer]

Can I get IR for the DISubprogram case?

[kinke]

It's the 'fixup' one earlier in this thread (#930 (comment)) - I don't think it has changed a lot. If need be I can recreate it after a recompile...

[majnemer]

Thanks, I switched to 0xffd8 and fixed the DISubprogram issue in r263408.

[kinke]

Thanks! I'll check it when I find some time.

[kinke]

I've tested it with yesterday's LLVM, works perfectly without any truncation from our side. Thx!