[COFF] Support MinGW automatic dllimport of data

Normally, in order to reference exported data symbols from a different
DLL, the declarations need to have the dllimport attribute, in order to
use the __imp_<var> symbol (which contains an address to the actual
variable) instead of the variable itself directly. This isn't an issue
in the same way for functions, since any reference to the function without
the dllimport attribute will end up as a reference to a thunk which loads
the actual target function from the import address table (IAT).

GNU ld, in MinGW environments, supports automatically importing data
symbols from DLLs, even if the references didn't have the appropriate
dllimport attribute. Since the PE/COFF format doesn't support the kind
of relocations that this would require, the MinGW's CRT startup code
has an custom framework of their own for manually fixing the missing
relocations once module is loaded and the target addresses in the IAT
are known.

For this to work, the linker (originall in GNU ld) creates a list of
remaining references needing fixup, which the runtime processes on
startup before handing over control to user code.

While this feature is rather controversial, it's one of the main features
allowing unix style libraries to be used on windows without any extra
porting effort.

Some sort of automatic fixing of data imports is also necessary for the
itanium C++ ABI on windows (as clang implements it right now) for importing
vtable pointers in certain cases, see D43184 for some discussion on that.

The runtime pseudo relocation handler supports 8/16/32/64 bit addresses,
either PC relative references (like IMAGE_REL_*_REL32*) or absolute
references (IMAGE_REL_AMD64_ADDR32, IMAGE_REL_AMD64_ADDR32,
IMAGE_REL_I386_DIR32). On linking, the relocation is handled as a
relocation against the corresponding IAT slot. For the absolute references,
a normal base relocation is created, to update the embedded address
in case the image is loaded at a different address.

The list of runtime pseudo relocations contains the RVA of the
imported symbol (the IAT slot), the RVA of the location the relocation
should be applied to, and a size of the memory location. When the
relocations are fixed at runtime, the difference between the actual
IAT slot value and the IAT slot address is added to the reference,
doing the right thing for both absolute and relative references.

With this patch alone, things work fine for i386 binaries, and mostly
for x86_64 binaries, with feature parity with GNU ld. Despite this,
there are a few gotchas:
- References to data from within code works fine on both x86 architectures,
  since their relocations consist of plain 32 or 64 bit absolute/relative
  references. On ARM and AArch64, references to data doesn't consist of
  a plain 32 or 64 bit embedded address or offset in the code. On ARMNT,
  it's usually a MOVW+MOVT instruction pair represented by a
  IMAGE_REL_ARM_MOV32T relocation, each instruction containing 16 bit of
  the target address), on AArch64, it's usually an ADRP+ADD/LDR/STR
  instruction pair with an even more complex encoding, storing a PC
  relative address (with a range of +/- 4 GB). This could theoretically
  be remedied by extending the runtime pseudo relocation handler with new
  relocation types, to support these instruction encodings. This isn't an
  issue for GCC/GNU ld since they don't support windows on ARMNT/AArch64.
- For x86_64, if references in code are encoded as 32 bit PC relative
  offsets, the runtime relocation will fail if the target turns out to be
  out of range for a 32 bit offset.
- Fixing up the relocations at runtime requires making sections writable
  if necessary, with the VirtualProtect function. In Windows Store/UWP apps,
  this function is forbidden.

These limitations are addressed by a few later patches in lld and
llvm.

Differential Revision: https://reviews.llvm.org/D50917

llvm-svn: 340726

Author: mstorsjo

lld/COFF/Chunks.cpp

@@ -421,6 +421,111 @@ void SectionChunk::getBaserels(std::vector<Baserel> *Res) {
   }
 }
 
+// MinGW specific.
+// Check whether a static relocation of type Type can be deferred and
+// handled at runtime as a pseudo relocation (for references to a module
+// local variable, which turned out to actually need to be imported from
+// another DLL) This returns the size the relocation is supposed to update,
+// in bits, or 0 if the relocation cannot be handled as a runtime pseudo
+// relocation.
+static int getRuntimePseudoRelocSize(uint16_t Type) {
+  // Relocations that either contain an absolute address, or a plain
+  // relative offset, since the runtime pseudo reloc implementation
+  // adds 8/16/32/64 bit values to a memory address.
+  //
+  // Given a pseudo relocation entry,
+  //
+  // typedef struct {
+  //   DWORD sym;
+  //   DWORD target;
+  //   DWORD flags;
+  // } runtime_pseudo_reloc_item_v2;
+  //
+  // the runtime relocation performs this adjustment:
+  //     *(base + .target) += *(base + .sym) - (base + .sym)
+  //
+  // This works for both absolute addresses (IMAGE_REL_*_ADDR32/64,
+  // IMAGE_REL_I386_DIR32, where the memory location initially contains
+  // the address of the IAT slot, and for relative addresses (IMAGE_REL*_REL32),
+  // where the memory location originally contains the relative offset to the
+  // IAT slot.
+  //
+  // This requires the target address to be writable, either directly out of
+  // the image, or temporarily changed at runtime with VirtualProtect.
+  // Since this only operates on direct address values, it doesn't work for
+  // ARM/ARM64 relocations, other than the plain ADDR32/ADDR64 relocations.
+  switch (Config->Machine) {
+  case AMD64:
+    switch (Type) {
+    case IMAGE_REL_AMD64_ADDR64:
+      return 64;
+    case IMAGE_REL_AMD64_ADDR32:
+    case IMAGE_REL_AMD64_REL32:
+    case IMAGE_REL_AMD64_REL32_1:
+    case IMAGE_REL_AMD64_REL32_2:
+    case IMAGE_REL_AMD64_REL32_3:
+    case IMAGE_REL_AMD64_REL32_4:
+    case IMAGE_REL_AMD64_REL32_5:
+      return 32;
+    default:
+      return 0;
+    }
+  case I386:
+    switch (Type) {
+    case IMAGE_REL_I386_DIR32:
+    case IMAGE_REL_I386_REL32:
+      return 32;
+    default:
+      return 0;
+    }
+  case ARMNT:
+    switch (Type) {
+    case IMAGE_REL_ARM_ADDR32:
+      return 32;
+    default:
+      return 0;
+    }
+  case ARM64:
+    switch (Type) {
+    case IMAGE_REL_ARM64_ADDR64:
+      return 64;
+    case IMAGE_REL_ARM64_ADDR32:
+      return 32;
+    default:
+      return 0;
+    }
+  default:
+    llvm_unreachable("unknown machine type");
+  }
+}
+
+// MinGW specific.
+// Append information to the provided vector about all relocations that
+// need to be handled at runtime as runtime pseudo relocations (references
+// to a module local variable, which turned out to actually need to be
+// imported from another DLL).
+void SectionChunk::getRuntimePseudoRelocs(
+    std::vector<RuntimePseudoReloc> &Res) {
+  for (const coff_relocation &Rel : Relocs) {
+    auto *Target = dyn_cast_or_null<DefinedImportData>(
+        File->getSymbol(Rel.SymbolTableIndex));
+    if (!Target || !Target->IsRuntimePseudoReloc)
+      continue;
+    int SizeInBits = getRuntimePseudoRelocSize(Rel.Type);
+    if (SizeInBits == 0) {
+      error("unable to automatically import from " + Target->getName() +
+            " with relocation type " +
+            File->getCOFFObj()->getRelocationTypeName(Rel.Type) + " in " +
+            toString(File));
+      continue;
+    }
+    // SizeInBits is used to initialize the Flags field; currently no
+    // other flags are defined.
+    Res.emplace_back(
+        RuntimePseudoReloc(Target, this, Rel.VirtualAddress, SizeInBits));
+  }
+}
+
 bool SectionChunk::hasData() const {
   return !(Header->Characteristics & IMAGE_SCN_CNT_UNINITIALIZED_DATA);
 }
@@ -539,6 +644,34 @@ void RVATableChunk::writeTo(uint8_t *Buf) const {
          "RVA tables should be de-duplicated");
 }
 
+// MinGW specific, for the "automatic import of variables from DLLs" feature.
+size_t PseudoRelocTableChunk::getSize() const {
+  if (Relocs.empty())
+    return 0;
+  return 12 + 12 * Relocs.size();
+}
+
+// MinGW specific.
+void PseudoRelocTableChunk::writeTo(uint8_t *Buf) const {
+  if (Relocs.empty())
+    return;
+
+  ulittle32_t *Table = reinterpret_cast<ulittle32_t *>(Buf + OutputSectionOff);
+  // This is the list header, to signal the runtime pseudo relocation v2
+  // format.
+  Table[0] = 0;
+  Table[1] = 0;
+  Table[2] = 1;
+
+  size_t Idx = 3;
+  for (const RuntimePseudoReloc &RPR : Relocs) {
+    Table[Idx + 0] = RPR.Sym->getRVA();
+    Table[Idx + 1] = RPR.Target->getRVA() + RPR.TargetOffset;
+    Table[Idx + 2] = RPR.Flags;
+    Idx += 3;
+  }
+}
+
 // Windows-specific. This class represents a block in .reloc section.
 // The format is described here.
 //

lld/COFF/Chunks.h

@@ -36,6 +36,7 @@ class DefinedImportData;
 class DefinedRegular;
 class ObjFile;
 class OutputSection;
+class RuntimePseudoReloc;
 class Symbol;
 
 // Mask for permissions (discardable, writable, readable, executable, etc).
@@ -161,6 +162,8 @@ class SectionChunk final : public Chunk {
   void applyRelARM64(uint8_t *Off, uint16_t Type, OutputSection *OS, uint64_t S,
                      uint64_t P) const;
 
+  void getRuntimePseudoRelocs(std::vector<RuntimePseudoReloc> &Res);
+
   // Called if the garbage collector decides to not include this chunk
   // in a final output. It's supposed to print out a log message to stdout.
   void printDiscardedMessage() const;
@@ -418,6 +421,53 @@ class Baserel {
   uint8_t Type;
 };
 
+// This is a placeholder Chunk, to allow attaching a DefinedSynthetic to a
+// specific place in a section, without any data. This is used for the MinGW
+// specific symbol __RUNTIME_PSEUDO_RELOC_LIST_END__, even though the concept
+// of an empty chunk isn't MinGW specific.
+class EmptyChunk : public Chunk {
+public:
+  EmptyChunk() {}
+  size_t getSize() const override { return 0; }
+  void writeTo(uint8_t *Buf) const override {}
+};
+
+// MinGW specific, for the "automatic import of variables from DLLs" feature.
+// This provides the table of runtime pseudo relocations, for variable
+// references that turned out to need to be imported from a DLL even though
+// the reference didn't use the dllimport attribute. The MinGW runtime will
+// process this table after loading, before handling control over to user
+// code.
+class PseudoRelocTableChunk : public Chunk {
+public:
+  PseudoRelocTableChunk(std::vector<RuntimePseudoReloc> &Relocs)
+      : Relocs(std::move(Relocs)) {
+    Alignment = 4;
+  }
+  size_t getSize() const override;
+  void writeTo(uint8_t *Buf) const override;
+
+private:
+  std::vector<RuntimePseudoReloc> Relocs;
+};
+
+// MinGW specific; information about one individual location in the image
+// that needs to be fixed up at runtime after loading. This represents
+// one individual element in the PseudoRelocTableChunk table.
+class RuntimePseudoReloc {
+public:
+  RuntimePseudoReloc(Defined *Sym, SectionChunk *Target, uint32_t TargetOffset,
+                     int Flags)
+      : Sym(Sym), Target(Target), TargetOffset(TargetOffset), Flags(Flags) {}
+
+  Defined *Sym;
+  SectionChunk *Target;
+  uint32_t TargetOffset;
+  // The Flags field contains the size of the relocation, in bits. No other
+  // flags are currently defined.
+  int Flags;
+};
+
 void applyMOV32T(uint8_t *Off, uint32_t V);
 void applyBranch24T(uint8_t *Off, int32_t V);
 

lld/COFF/Driver.cpp

@@ -1367,6 +1367,11 @@ void LinkerDriver::link(ArrayRef<const char *> ArgsArr) {
   // Needed for MSVC 2017 15.5 CRT.
   Symtab->addAbsolute(mangle("__enclave_config"), 0);
 
+  if (Config->MinGW) {
+    Symtab->addAbsolute(mangle("__RUNTIME_PSEUDO_RELOC_LIST__"), 0);
+    Symtab->addAbsolute(mangle("__RUNTIME_PSEUDO_RELOC_LIST_END__"), 0);
+  }
+
   // This code may add new undefined symbols to the link, which may enqueue more
   // symbol resolution tasks, so we need to continue executing tasks until we
   // converge.
@@ -1411,6 +1416,24 @@ void LinkerDriver::link(ArrayRef<const char *> ArgsArr) {
   Symtab->addCombinedLTOObjects();
   run();
 
+  if (Config->MinGW) {
+    // Load any further object files that might be needed for doing automatic
+    // imports.
+    //
+    // For cases with no automatically imported symbols, this iterates once
+    // over the symbol table and doesn't do anything.
+    //
+    // For the normal case with a few automatically imported symbols, this
+    // should only need to be run once, since each new object file imported
+    // is an import library and wouldn't add any new undefined references,
+    // but there's nothing stopping the __imp_ symbols from coming from a
+    // normal object file as well (although that won't be used for the
+    // actual autoimport later on). If this pass adds new undefined references,
+    // we won't iterate further to resolve them.
+    Symtab->loadMinGWAutomaticImports();
+    run();
+  }
+
   // Make sure we have resolved all symbols.
   Symtab->reportRemainingUndefines();
   if (errorCount())

lld/COFF/SymbolTable.cpp

@@ -125,6 +125,53 @@ static std::string getSymbolLocations(ObjFile *File, uint32_t SymIndex) {
   return OS.str();
 }
 
+void SymbolTable::loadMinGWAutomaticImports() {
+  for (auto &I : SymMap) {
+    Symbol *Sym = I.second;
+    auto *Undef = dyn_cast<Undefined>(Sym);
+    if (!Undef)
+      continue;
+    if (!Sym->IsUsedInRegularObj)
+      continue;
+
+    StringRef Name = Undef->getName();
+
+    if (Name.startswith("__imp_"))
+      continue;
+    // If we have an undefined symbol, but we have a Lazy representing a
+    // symbol we could load from file, make sure to load that.
+    Lazy *L = dyn_cast_or_null<Lazy>(find(("__imp_" + Name).str()));
+    if (!L || L->PendingArchiveLoad)
+      continue;
+
+    log("Loading lazy " + L->getName() + " from " + L->File->getName() +
+        " for automatic import");
+    L->PendingArchiveLoad = true;
+    L->File->addMember(&L->Sym);
+  }
+}
+
+bool SymbolTable::handleMinGWAutomaticImport(Symbol *Sym, StringRef Name) {
+  if (Name.startswith("__imp_"))
+    return false;
+  DefinedImportData *Imp =
+      dyn_cast_or_null<DefinedImportData>(find(("__imp_" + Name).str()));
+  if (!Imp)
+    return false;
+
+  log("Automatically importing " + Name + " from " + Imp->getDLLName());
+
+  // Replace the reference directly to a variable with a reference
+  // to the import address table instead. This obviously isn't right,
+  // but we mark the symbol as IsRuntimePseudoReloc, and a later pass
+  // will add runtime pseudo relocations for every relocation against
+  // this Symbol. The runtime pseudo relocation framework expects the
+  // reference itself to point at the IAT entry.
+  Sym->replaceKeepingName(Imp, sizeof(DefinedImportData));
+  cast<DefinedImportData>(Sym)->IsRuntimePseudoReloc = true;
+  return true;
+}
+
 void SymbolTable::reportRemainingUndefines() {
   SmallPtrSet<Symbol *, 8> Undefs;
   DenseMap<Symbol *, Symbol *> LocalImports;
@@ -168,6 +215,9 @@ void SymbolTable::reportRemainingUndefines() {
       }
     }
 
+    if (Config->MinGW && handleMinGWAutomaticImport(Sym, Name))
+      continue;
+
     // Remaining undefined symbols are not fatal if /force is specified.
     // They are replaced with dummy defined symbols.
     if (Config->Force)

lld/COFF/SymbolTable.h

@@ -54,6 +54,9 @@ class SymbolTable {
   // symbols.
   void reportRemainingUndefines();
 
+  void loadMinGWAutomaticImports();
+  bool handleMinGWAutomaticImport(Symbol *Sym, StringRef Name);
+
   // Returns a list of chunks of selected symbols.
   std::vector<Chunk *> getChunks();
 

lld/COFF/Symbols.cpp

@@ -63,6 +63,13 @@ bool Symbol::isLive() const {
   return true;
 }
 
+// MinGW specific.
+void Symbol::replaceKeepingName(Symbol *Other, size_t Size) {
+  StringRef OrigName = Name;
+  memcpy(this, Other, Size);
+  Name = OrigName;
+}
+
 COFFSymbolRef DefinedCOFF::getCOFFSymbol() {
   size_t SymSize = cast<ObjFile>(File)->getCOFFObj()->getSymbolTableEntrySize();
   if (SymSize == sizeof(coff_symbol16))

lld/COFF/Symbols.h

@@ -66,6 +66,8 @@ class Symbol {
   // Returns the symbol name.
   StringRef getName();
 
+  void replaceKeepingName(Symbol *Other, size_t Size);
+
   // Returns the file from which this symbol was created.
   InputFile *getFile();
 
@@ -307,6 +309,8 @@ class DefinedImportData : public Defined {
   uint16_t getOrdinal() { return File->Hdr->OrdinalHint; }
 
   ImportFile *File;
+
+  bool IsRuntimePseudoReloc = false;
 };
 
 // This class represents a symbol for a jump table entry which jumps