Sanitizer handler calls emitted without regard to -mregparm

[kees]

When sanitizer calls are emitted, the -mregparm=3 option used by the Linux kernel appears to be ignored. For example, here is a build where the argument are being pushed instead of placed in %eax and %edx (from lkdtm_ARRAY_BOUNDS):

   0xc18e3a5a <+202>:   push   %ebx
   0xc18e3a5b <+203>:   push   $0xc26001a0
   0xc18e3a60 <+208>:   call   0xc157d430 <__ubsan_handle_out_of_bounds>

The kernel's handler isn't expecting them on the stack. For example, this is setting a bit in the sanitizer's passed-in data structure (from __ubsan_handle_out_of_bounds):

   0xc157d491 <+97>:    btsl   $0x1f,%ds:0x4(%eax)
   0xc157d497 <+103>:   jae    0xc157d4a1 <__ubsan_handle_out_of_bounds+113>

KSPP/linux#350

[kees]

From clang/lib/CodeGen/CGCall.cpp:

llvm::CallBase *
CodeGenFunction::EmitRuntimeCallOrInvoke(llvm::FunctionCallee callee,
                                         ArrayRef<llvm::Value *> args,
                                         const Twine &name) {
  llvm::CallBase *call = EmitCallOrInvoke(callee, args, name);
  call->setCallingConv(getRuntimeCC());
  return call;
}

the calling convention doesn't seem to have any knowledge of where -mregparm gets recorded in clang/lib/CodeGen/CodeGenModule.cpp from CodeGenModule::Release():

  // Record mregparm value now so it is visible through rest of codegen.
  if (Context.getTargetInfo().getTriple().getArch() == llvm::Triple::x86)
    getModule().addModuleFlag(llvm::Module::Error, "NumRegisterParameters",
                              CodeGenOpts.NumRegisterParameters);

[llvmbot]

@llvm/issue-subscribers-clang-codegen

Author: Kees Cook (kees)

When sanitizer calls are emitted, the `-mregparm=3` option used by the Linux kernel appears to be ignored. For example, here is a build where the argument are being pushed instead of placed in `%eax` and `%edx` (from `lkdtm_ARRAY_BOUNDS`):

   0xc18e3a5a <+202>:   push   %ebx
   0xc18e3a5b <+203>:   push   $0xc26001a0
   0xc18e3a60 <+208>:   call   0xc157d430 <__ubsan_handle_out_of_bounds>

The kernel's handler isn't expecting them on the stack. For example, this is setting a bit in the sanitizer's passed-in data structure (from __ubsan_handle_out_of_bounds):

   0xc157d491 <+97>:    btsl   $0x1f,%ds:0x4(%eax)
   0xc157d497 <+103>:   jae    0xc157d4a1 <__ubsan_handle_out_of_bounds+113>

KSPP/linux#350