[cdac] cdac-build-tool (dotnet#100650)

# cDAC Build Tool

## Summary

The purpose of `cdac-build-tool` is to generate a `.c` file that contains a JSON cDAC contract descriptor.

It works by processing one or more object files containing data descriptors and zero or more text
files that specify contracts.

## Running

```console
% cdac-build-tool compose [-v] -o contractdescriptor.c -c contracts.txt datadescriptor.o
```
## .NET runtime build integration

`cdac-build-tool` is meant to run as a CMake custom command.
It consumes a target platform object file and emits a C source
file that contains a JSON contract descriptor.  The C source
is the included in the normal build and link steps to create the runtime.

The contract descriptor source file depends on `contract-aux-data.c` which is a source file that contains
the definitions of the "indirect pointer data" that is referenced by the data descriptor.  This is typically the addresses of important global variables in the runtime.
Constants and build flags are embedded directly in the JSON payload.

Multiple data descriptor source files may be specified (for example if they are produced by different components of the runtime, or by different source languages).  The final JSON payload will be a composition of all the data descriptors.

Multiple contracts text files may be specified.  This may be useful if some contracts are conditionally included (for example if they are platform-specific).  The final JSON payload will be a composition of all the contracts files.

```mermaid
flowchart TB
  headers("runtime headers")
  data_header("datadescriptor.h")
  data_src("datadescriptor.c")
  compile_data["clang"]
  data_obj("datadescriptor.o")
  contracts("contracts.txt")
  globals("contractpointerdata.c")
  build[["cdac-build-tool"]]
  descriptor_src("contractdescriptor.c")
  vm("runtime sources")
  compile_runtime["clang"]
  runtime_lib(["libcoreclr.so"])

  headers -.-> data_src
  headers ~~~ data_header
  data_header -.-> data_src
  headers -.-> globals
  headers -.-> vm
  data_src --> compile_data --> data_obj --> build
  contracts ---> build
  build --> descriptor_src
  descriptor_src --> compile_runtime
  data_header -.-> globals ----> compile_runtime
  vm ----> compile_runtime --> runtime_lib
```


--- 

* add implementation note notes

* add an emitter

* read in the directory header

* contract parsing

* indirect pointer value support

* move sample to tool dir

* Take baselines from the docs/design/datacontracts/data dir

  We don't parse them yet, however

* Add README

* fix BE

   Store the magic as a uint64_t so that it will follow the platform endianness.

   Store endmagic as bytes so that it directly follows the name pool - and fix the endmagic check not to look at the endianness

* hook up cdac-build-tool to the coreclr build; export DotNetRuntimeContractDescriptor

* cleanup; add contracts.txt

* add diagram to README

* move implementation notes

* better verbose output from ObjectFileScraper

* turn off whole program optimizations for data-descriptor.obj

   On windows /GL creates object files that cdac-build-tool cannot read

   It's ok to do this because we don't ship data-descriptor.obj as part of the product - it's only used to generate the cDAC descriptor

* C++-ify and add real Thread offsets

* no C99 designated initializers in C++ until C++20

* build data descriptor after core runtime

* fix gcc build

* simplify ObjectFileScraper

   just read the whole file into memory

* invoke 'dotnet cmake-build-tool.dll' instead of 'dotnet run --project'

* clean up macro boilerplate

* platform flags

* turn off verbose output

* can't use constexpr function in coreclr

   because debugreturn.h defines a `return` macro that expands to something that is not c++11 constexpr

* Rename "aux data" to "pointer data"

* rename "data-descriptor" to "datadescriptor"

* simplify linking

* cdac-build-tool don't build dotnet tool; turn on analyzers

* rationalize naming; update docs; add some inline comments

* renamce cdac.h to cdacoffsets.h

* improve output: hex offsets; improved formatting

* don't throw in ParseContracts; add line numbers to errors

* change input format for contracts to jsonc

* add custom JsonConverter instances for the compact json representation

* simplify; bug fix - PointerDataCount include placeholder

* one more set of feedback changes: simpler json converters

* set _RequiresLiveILLink=false for cdac-build-tool.csproj

   fixes windows builds:

   error MSB3026: (NETCORE_ENGINEERING_TELEMETRY=Build) Could not copy "D:\a\_work\1\s\artifacts\obj\ILLink.Tasks\Debug\net9.0\ILLink.Tasks.dll" to "D:\a\_work\1\s\artifacts\bin\ILLink.Tasks\Debug\net9.0\ILLink.Tasks.dll". Beginning retry 1 in 1000ms. The process cannot access the file 'D:\a\_work\1\s\artifacts\bin\ILLink.Tasks\Debug\net9.0\ILLink.Tasks.dll' because it is being used by another process. 

   
---------

Co-authored-by: Elinor Fung <elfung@microsoft.com>
Co-authored-by: Aaron Robinson <arobins@microsoft.com>

Author: 3 people

docs/design/datacontracts/contract-descriptor.md

@@ -24,9 +24,9 @@ struct DotNetRuntimeContractDescriptor
     uint32_t flags;
     uint32_t descriptor_size;
     const char *descriptor;
-    uint32_t aux_data_count;
+    uint32_t pointer_data_count;
     uint32_t pad0;
-    uintptr_t *aux_data;
+    uintptr_t *pointer_data;
 };
 ```
 

docs/design/datacontracts/data/empty.jsonc

@@ -0,0 +1,4 @@
+// the empty baseline data descriptor
+{
+    "version": 0
+}

eng/Subsets.props

@@ -360,6 +360,8 @@
     <ProjectToBuild Include="$(CoreClrProjectRoot)tools\aot\crossgen2\crossgen2_inbuild.csproj" Category="clr" />
 
     <ProjectToBuild Condition="'$(TargetOS)' == 'windows' or ('$(TargetOS)' == 'linux' and ('$(TargetArchitecture)' == 'x64' or '$(TargetArchitecture)' == 'arm64')) or '$(TargetOS)' == 'osx'" Include="$(CoreClrProjectRoot)tools\SuperFileCheck\SuperFileCheck.csproj" Category="clr" />
+
+    <ProjectToBuild Include="$(CoreClrProjectRoot)tools\cdac-build-tool\cdac-build-tool.csproj" Category="clr" />
   </ItemGroup>
 
   <ItemGroup Condition="$(_subset.Contains('+clr.toolstests+'))">

src/coreclr/debug/runtimeinfo/CMakeLists.txt

@@ -37,3 +37,46 @@ endif()
 
 # publish runtimeinfo lib
 install_clr(TARGETS runtimeinfo DESTINATIONS lib COMPONENT runtime)
+
+add_library(cdac_data_descriptor OBJECT datadescriptor.cpp)
+# don't build the data descriptor before the VM (and any of its dependencies' generated headers)
+add_dependencies(cdac_data_descriptor cee_wks_core)
+if(CLR_CMAKE_TARGET_WIN32)
+  # turn off whole program optimization:
+  # 1. it creates object files that cdac-build-tool can't read
+  # 2. we never link cdac_data_descriptor into the final product - it's only job is to be scraped
+  target_compile_options(cdac_data_descriptor PRIVATE /GL-)
+endif()
+target_include_directories(cdac_data_descriptor BEFORE PRIVATE ${VM_DIR})
+target_include_directories(cdac_data_descriptor BEFORE PRIVATE ${VM_DIR}/${ARCH_SOURCES_DIR})
+target_include_directories(cdac_data_descriptor PRIVATE ${CLR_DIR}/interop/inc)
+
+set(GENERATED_CDAC_DESCRIPTOR_DIR "${CMAKE_CURRENT_BINARY_DIR}/cdac")
+set(CONTRACT_DESCRIPTOR_OUTPUT "${GENERATED_CDAC_DESCRIPTOR_DIR}/contract-descriptor.c")
+if("${CDAC_BUILD_TOOL_BINARY_PATH}" STREQUAL "" OR NOT EXISTS "${CDAC_BUILD_TOOL_BINARY_PATH}")
+  message(FATAL_ERROR "No cdac-build-tool set or ${CDAC_BUILD_TOOL_BINARY_PATH} does not exist")
+endif()
+
+set(CONTRACT_FILE "${CMAKE_CURRENT_SOURCE_DIR}/contracts.jsonc")
+
+# generate the contract descriptor by running cdac-build-tool
+# n.b. this just uses `dotnet` from the PATH.  InitializeDotNetCli adds the apropropriate directory
+add_custom_command(
+    OUTPUT "${CONTRACT_DESCRIPTOR_OUTPUT}"
+    VERBATIM
+    COMMAND dotnet ${CDAC_BUILD_TOOL_BINARY_PATH} compose -o "${CONTRACT_DESCRIPTOR_OUTPUT}" -c "${CONTRACT_FILE}" $<TARGET_OBJECTS:cdac_data_descriptor>
+    DEPENDS cdac_data_descriptor cee_wks_core $<TARGET_OBJECTS:cdac_data_descriptor> "${CONTRACT_FILE}"
+    USES_TERMINAL
+)
+
+# It is important that cdac_contract_descriptor is an object library;
+# if it was static, linking it into the final dll would not export
+# DotNetRuntimeContractDescriptor since it is not referenced anywhere.
+add_library_clr(cdac_contract_descriptor OBJECT
+    "${CONTRACT_DESCRIPTOR_OUTPUT}"
+    contractpointerdata.cpp
+)
+target_include_directories(cdac_contract_descriptor BEFORE PRIVATE ${VM_DIR})
+target_include_directories(cdac_contract_descriptor BEFORE PRIVATE ${VM_DIR}/${ARCH_SOURCES_DIR})
+target_include_directories(cdac_contract_descriptor PRIVATE ${CLR_DIR}/interop/inc)
+add_dependencies(cdac_contract_descriptor cdac_data_descriptor cee_wks_core)

src/coreclr/debug/runtimeinfo/contractpointerdata.cpp

@@ -0,0 +1,23 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+#include "common.h"
+
+#include <stddef.h>
+#include <stdint.h>
+
+#include "threads.h"
+
+extern "C"
+{
+  
+// without an extern declaration, clang does not emit this global into the object file
+extern const uintptr_t contractDescriptorPointerData[];
+
+const uintptr_t contractDescriptorPointerData[] = {
+    (uintptr_t)0, // placeholder
+#define CDAC_GLOBAL_POINTER(name,value) (uintptr_t)(value),
+#include "datadescriptor.h"
+};
+
+}

src/coreclr/debug/runtimeinfo/contracts.jsonc

@@ -0,0 +1,14 @@
+//algorithmic contracts for coreclr
+// The format of this file is: JSON with comments
+//  {
+//     "CONTRACT NAME": VERSION,
+//     ...
+//   }
+// CONTRACT NAME is an arbitrary string, VERSION is an integer
+//
+// cdac-build-tool can take multiple "-c contract_file" arguments
+// so to conditionally include contracts, put additional contracts in a separate file
+{
+  "SOSBreakingChangeVersion": 1 // example contract: "runtime exports an SOS breaking change version global"
+}
+