Merge branch 'EpicGamesExt:master' into master

.github/workflows/builds.yml

@@ -20,9 +20,11 @@ jobs:
         shell: cmd
         run: |
           call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64
-          call build raddbg msvc debug             || exit /b 1
-          call build raddbg_from_pdb msvc debug    || exit /b 1
-          call build raddbg_from_dwarf msvc debug  || exit /b 1
-          call build raddbg clang debug            || exit /b 1
-          call build raddbg_from_pdb clang debug   || exit /b 1
-          call build raddbg_from_dwarf clang debug || exit /b 1
+          call build raddbg msvc debug              || exit /b 1
+          call build raddbgi_from_pdb msvc debug    || exit /b 1
+          call build raddbgi_from_dwarf msvc debug  || exit /b 1
+          call build raddbgi_dump msvc debug        || exit /b 1
+          call build raddbg clang debug             || exit /b 1
+          call build raddbgi_from_pdb clang debug   || exit /b 1
+          call build raddbgi_from_dwarf clang debug || exit /b 1
+          call build raddbgi_dump clang debug       || exit /b 1

README.md

@@ -1,5 +1,11 @@
 # The RAD Debugger Project
 
+_**Note:** This README does not document usage instructions and tips for the
+debugger itself, and is intended as a technical overview of the project. The
+debugger's README, which includes usage instructions and tips, can be found
+packaged along with debugger releases, or within the `build` folder after a
+local copy has been built._
+
 The RAD Debugger is a native, user-mode, multi-process, graphical debugger. It
 currently only supports local-machine Windows x64 debugging with PDBs, with
 plans to expand and port in the future. In the future we'll expand to also
@@ -15,22 +21,25 @@ You can download pre-built binaries for the debugger
 
 The RAD Debugger project aims to simplify the debugger by simplifying and
 unifying the underlying debug info format. In that pursuit we've built
-the RADDBG debug info format, which is what the debugger parses and uses. To
+the RADDBGI debug info format, which is what the debugger parses and uses. To
 work with existing toolchains, we convert PDB (and eventually PE/ELF files
-with embedded DWARF) into the RADDBG format on-demand. This conversion process
+with embedded DWARF) into the RADDBGI format on-demand. This conversion process
 is currently an unoptimized reference version. Nevertheless it's still quite
 fast for smaller PDB files (in many cases faster than many other programs
 simply deserialize the PDBs). It is much slower for much larger projects at the
 moment, but we expect this will vastly improve overtime.
 
-The RADDBG format is currently specified in code, in the files within the
-`src/raddbg_format` folder. The other relevant folders for working with the
-format are:
+The RADDBGI format is currently specified in code, in the files within the
+`src/lib_raddbgi_format` folder. The other relevant folders for working with
+the format are:
 
-- `raddbg_cons`: The RADDBG construction layer, for constructing RADDBG files.
-- `raddbg_convert`: Our implementation of PDB-to-RADDBG (and an in-progress
-implementation of a DWARF-to-RADDBG) conversion.
-- `raddbg_dump`: Code for textually dumping information from RADDBG files.
+- `lib_raddbgi_make`: The "RAD Debug Info Make" library, for making RADDBGI
+  debug info.
+- `raddbgi_from_pdb`: Our PDB-to-RADDBGI converter. Can be used as a helper
+  codebase layer, or built as an executable with a command line interface
+  frontend.
+- `raddbgi_from_dwarf`: Our in-progress DWARF-to-RADDBGI converter.
+- `raddbgi_dump`: Our RADDBGI textual dumping utility.
 
 ## Development Setup Instructions
 
@@ -119,13 +128,13 @@ such that the debugging experience is rock solid.
 
 Additionally, the debug info conversion process is not fast (nor wide) enough
 to support extremely large projects. This is for two reasons: (a) the
-PDB-to-RADDBG converter is an unoptimized reference implementation, and (b) the
-debugger learns of new modules (and thus which PDBs to load) in a
+PDB-to-RADDBGI converter is an unoptimized reference implementation, and (b)
+the debugger learns of new modules (and thus which PDBs to load) in a
 serially-dependent way (this is necessarily the case for correct debugging
 results). We expect that the conversion process' performance can be massively
 improved, and also that some heuristics can be used to begin converting PDBs
-to RADDBGs before the debugger knows those PDBs are needed, thus ensuring the
-associated RADDBG files are ready instantaneously when the associated modules
+to RADDBGIs before the debugger knows those PDBs are needed, thus ensuring the
+associated RADDBGI files are ready instantaneously when the associated modules
 are finally loaded by the debugger. Improving this situation is a major part of
 this phase, as it will make the debugger much more usable for large projects.
 
@@ -145,9 +154,9 @@ The major parts of this phase are:
 abstraction API.
 - Porting the `src/unwind` layer to support x64 ELF unwinding (currently, there
 is only an x64 PE unwinding implementation).
-- Creating a DWARF-to-RADDBG converter (in the same way that we've built a PDB-
-to-RADDBG converter). A partial implementation of this is in
-`src/raddbg_convert/dwarf`.
+- Creating a DWARF-to-RADDBGI converter (in the same way that we've built a
+PDB-to-RADDBGI converter). A partial implementation of this is in
+`src/raddbgi_from_dwarf`.
 - Porting the `src/render` layer to implement all of the rendering features the
 frontend needs on a Linux-compatible API (the backend used on Windows is D3D11).
 - Porting the `src/font_provider` layer to a Linux-compatible font
@@ -206,10 +215,16 @@ Layers depend on other layers, but circular dependencies would break the
 separability and isolation utility of layers (in effect, forming one big layer),
 so in other words, layers are arranged into a directed acyclic graph.
 
+A few layers are built to be used completely independently from the rest of the
+codebase, as libraries in other codebases and projects. As such, these layers do
+not depend on any other layers in the codebase. The folders which contain these
+layers are prefixed with `lib_`, like `lib_raddbgi_format`.
+
 A list of the layers in the codebase and their associated namespaces is below:
 - `base` (no namespace): Universal, codebase-wide constructs. Strings, math,
   memory allocators, helper macros, command-line parsing, and so on. Depends
   on no other codebase layers.
+- `codeview` (`CV_`): Code for parsing and/or writing the CodeView format.
 - `coff` (`COFF_`): Code for parsing and/or writing the COFF (Common Object File
   Format) file format.
 - `ctrl` (`CTRL_`): The debugger's "control system" layer. Implements
@@ -221,10 +236,10 @@ A list of the layers in the codebase and their associated namespaces is below:
   disassembly for a particular virtual address range in a process, and threads
   implemented in this layer decode and cache the disassembly for that range.
 - `dbgi` (`DBGI_`): An asynchronous debug info loader and cache. Loads debug
-  info stored in the RADDBG format. Users ask for debug info for a particular
+  info stored in the RADDBGI format. Users ask for debug info for a particular
   executable, and on separate threads, this layer loads the associated debug
   info file. If necessary, it will launch a separate conversion process to
-  convert original debug info into the RADDBG format.
+  convert original debug info into the RADDBGI format.
 - `demon` (`DEMON_`): An abstraction layer for local-machine, low-level process
   control. The abstraction is used to provide a common interface for process
   control on target platforms. Used to implement part of `ctrl`.
@@ -256,6 +271,16 @@ A list of the layers in the codebase and their associated namespaces is below:
   for asynchronously preparing data for memory visualization in the debugger.
 - `hash_store` (`HS_`): Implements a cache for general data blobs, keyed by a
   128-bit hash of the data. Used as a general data store by other layers.
+- `lib_raddbg_markup` (`RADDBG_`): Standalone library for marking up user
+  programs to work with various features in the `raddbg` debugger. Does not
+  depend on `base`, and can be independently relocated to other codebases.
+- `lib_raddbgi_make` (`RDIM_`): Standalone library for constructing RADDBGI
+  debug info data. Does not depend on `base`, and can be independently relocated
+  to other codebases.
+- `lib_raddbgi_format` (`RDI_`): Standalone library which defines the core
+  RADDBGI types and helper functions for reading and writing the RADDBGI debug
+  info file format. Does not depend on `base`, and can be independently
+  relocated to other codebases.
 - `metagen` (`MG_`): A metaprogram which is used to generate primarily code and
   data tables. Consumes Metadesk files, stored with the extension `.mdesk`, and
   generates C code which is then included by hand-written C code. Currently, it
@@ -270,6 +295,7 @@ A list of the layers in the codebase and their associated namespaces is below:
   duplicate version of `base` and `os` are included in this layer. They are
   updated manually, as needed. This is to ensure the stability of the
   metaprogram.
+- `msf` (`MSF_`): Code for parsing and/or writing the MSF file format.
 - `mule` (no namespace): Test executables for battle testing debugger
   functionality.
 - `natvis` (no namespace): NatVis files for type visualization of the codebase's
@@ -283,20 +309,17 @@ A list of the layers in the codebase and their associated namespaces is below:
 - `os/socket` (`OS_`): An abstraction layer, building on `os/core`, providing
   networking operating system features under an abstract API, which is
   implemented per-target-operating-system.
+- `pdb` (`PDB_`): Code for parsing and/or writing the PDB file format.
 - `pe` (`PE_`): Code for parsing and/or writing the PE (Portable Executable)
   file format.
 - `raddbg` (no namespace): The layer which ties everything together for the main
   graphical debugger. Not much "meat", just drives `df`, implements command line
   options, and so on.
-- `raddbg_cons` (`CONS_`): Implements an API for constructing files of the
-  RADDBG debug info file format.
-- `raddbg_dump` (`DUMP_`): A dumper utility program for dumping textualizations
-  of RADDBG debug info files.
-- `raddbg_format` (`RADDBG_`): Standalone types and helper functions for the
-  RADDBG debug info file format. Does not depend on `base`.
-- `raddbg_markup` (`RADDBG_`): Standalone header file for marking up user
-  programs to work with various features in the `raddbg` debugger. Does not
-  depend on `base`.
+- `raddbgi_from_pdb` (`P2R_`): Our implementation of PDB-to-RADDBGI conversion.
+- `raddbgi_from_dwarf` (`D2R_`): Our in-progress implementation of
+  DWARF-to-RADDBGI conversion.
+- `raddbgi_dump` (`RADDBGIDUMP_`): A dumper utility program for dumping
+  textualizations of RADDBGI debug info files.
 - `regs` (`REGS_`): Types, helper functions, and metadata for registers on
   supported architectures. Used in reading/writing registers in `demon`, or in
   looking up register metadata.
@@ -315,7 +338,7 @@ A list of the layers in the codebase and their associated namespaces is below:
   Used by the debugger to visualize source code files. Users ask for text lines,
   tokens, and metadata, and it is prepared on background threads.
 - `type_graph` (`TG_`): Code for analyzing and navigating type structures from
-  RADDBG debug info files, with the additional capability of constructing
+  RADDBGI debug info files, with the additional capability of constructing
   synthetic types *not* found in debug info. Used in `eval` and for various
   visualization features.
 - `ui` (`UI_`): Machinery for building graphical user interfaces. Provides a

build.bat

@@ -15,7 +15,7 @@ cd /D "%~dp0"
 :: `build raddbg clang`
 :: `build raddbg release`
 :: `build raddbg asan telemetry`
-:: `build raddbg_from_pdb`
+:: `build raddbgi_from_pdb`
 ::
 :: For a full list of possible build targets and their build command lines,
 :: search for @build_targets in this file.
@@ -43,18 +43,16 @@ if "%asan%"=="1"      set auto_compile_flags=%auto_compile_flags% -fsanitize=add
 :: --- Compile/Link Line Definitions ------------------------------------------
 set cl_common=     /I..\src\ /I..\local\ /nologo /FC /Z7
 set clang_common=  -I..\src\ -I..\local\ -gcodeview -fdiagnostics-absolute-paths -Wall -Wno-unknown-warning-option -Wno-missing-braces -Wno-unused-function -Wno-writable-strings -Wno-unused-value -Wno-unused-variable -Wno-unused-local-typedef -Wno-deprecated-register -Wno-deprecated-declarations -Wno-unused-but-set-variable -Wno-single-bit-bitfield-constant-conversion -Xclang -flto-visibility-public-std -D_USE_MATH_DEFINES -Dstrdup=_strdup -Dgnu_printf=printf
-set cl_debug=      call cl /Od %cl_common% %auto_compile_flags%
-set cl_release=    call cl /O2 /DNDEBUG %cl_common% %auto_compile_flags%
-set clang_debug=   call clang -g -O0 %clang_common% %auto_compile_flags%
-set clang_release= call clang -g -O2 -DNDEBUG %clang_common% %auto_compile_flags%
+set cl_debug=      call cl /Od /DBUILD_DEBUG=1 %cl_common% %auto_compile_flags%
+set cl_release=    call cl /O2 /DBUILD_DEBUG=0 %cl_common% %auto_compile_flags%
+set clang_debug=   call clang -g -O0 -DBUILD_DEBUG=1 %clang_common% %auto_compile_flags%
+set clang_release= call clang -g -O2 -DBUILD_DEBUG=0 %clang_common% %auto_compile_flags%
 set cl_link=       /link /MANIFEST:EMBED /INCREMENTAL:NO /natvis:"%~dp0\src\natvis\base.natvis" logo.res
 set clang_link=    -fuse-ld=lld -Xlinker /MANIFEST:EMBED -Xlinker /natvis:"%~dp0\src\natvis\base.natvis" logo.res
 set cl_out=        /out:
 set clang_out=     -o
 
 :: --- Per-Build Settings -----------------------------------------------------
-set gfx=-DOS_FEATURE_GRAPHICAL=1
-set net=-DOS_FEATURE_SOCKET=1
 set link_dll=-DLL
 if "%msvc%"=="1"  set only_compile=/c
 if "%clang%"=="1" set only_compile=-c
@@ -85,7 +83,7 @@ pushd build
 popd
 
 :: --- Get Current Git Commit Id ----------------------------------------------
-for /f %%i in ('call git describe --always --dirty') do set compile=%compile% -DRADDBG_GIT=\"%%i\"
+for /f %%i in ('call git describe --always --dirty') do set compile=%compile% -DBUILD_GIT_HASH=\"%%i\"
 
 :: --- Build & Run Metaprogram ------------------------------------------------
 if "%no_meta%"=="1" echo [skipping metagen]
@@ -98,16 +96,17 @@ if not "%no_meta%"=="1" (
 
 :: --- Build Everything (@build_targets) --------------------------------------
 pushd build
-if "%raddbg%"=="1"             %compile% %gfx%       ..\src\raddbg\raddbg_main.cpp                                %compile_link% %out%raddbg.exe || exit /b 1
-if "%raddbg_from_pdb%"=="1"    %compile%             ..\src\raddbg_convert\pdb\raddbg_from_pdb_main.c             %compile_link% %out%raddbg_from_pdb.exe || exit /b 1
-if "%raddbg_from_dwarf%"=="1"  %compile%             ..\src\raddbg_convert\dwarf\raddbg_from_dwarf.c              %compile_link% %out%raddbg_from_dwarf.exe || exit /b 1
-if "%raddbg_dump%"=="1"        %compile%             ..\src\raddbg_dump\raddbg_dump.c                             %compile_link% %out%raddbg_dump.exe || exit /b 1
-if "%ryan_scratch%"=="1"       %compile%             ..\src\scratch\ryan_scratch.c                                %compile_link% %out%ryan_scratch.exe || exit /b 1
-if "%cpp_tests%"=="1"          %compile%             ..\src\scratch\i_hate_c_plus_plus.cpp                        %compile_link% %out%cpp_tests.exe || exit /b 1
-if "%look_at_raddbg%"=="1"     %compile%             ..\src\scratch\look_at_raddbg.c                              %compile_link% %out%look_at_raddbg.exe || exit /b 1
-if "%mule_main%"=="1"          del vc*.pdb mule*.pdb && %compile_release% %only_compile% ..\src\mule\mule_inline.cpp && %compile_release% %only_compile% ..\src\mule\mule_o2.cpp && %compile_debug% %EHsc% ..\src\mule\mule_main.cpp ..\src\mule\mule_c.c mule_inline.obj mule_o2.obj %compile_link% %out%mule_main.exe || exit /b 1
-if "%mule_module%"=="1"        %compile%             ..\src\mule\mule_module.cpp                                  %compile_link% %link_dll% %out%mule_module.dll || exit /b 1
-if "%mule_hotload%"=="1"       %compile% ..\src\mule\mule_hotload_main.c %compile_link% %out%mule_hotload.exe & %compile% ..\src\mule\mule_hotload_module_main.c %compile_link% %link_dll% %out%mule_hotload_module.dll || exit /b 1
+if "%raddbg%"=="1"                     %compile% %gfx%       ..\src\raddbg\raddbg_main.cpp                                                %compile_link% %out%raddbg.exe || exit /b 1
+if "%raddbgi_from_pdb%"=="1"           %compile%             ..\src\raddbgi_from_pdb\raddbgi_from_pdb_main.c                              %compile_link% %out%raddbgi_from_pdb.exe || exit /b 1
+if "%raddbgi_from_dwarf%"=="1"         %compile%             ..\src\raddbgi_from_dwarf\raddbgi_from_dwarf.c                               %compile_link% %out%raddbgi_from_dwarf.exe || exit /b 1
+if "%raddbgi_dump%"=="1"               %compile%             ..\src\raddbgi_dump\raddbgi_dump_main.c                                      %compile_link% %out%raddbgi_dump.exe || exit /b 1
+if "%raddbgi_breakpad_from_pdb%"=="1"  %compile%             ..\src\raddbgi_breakpad_from_pdb\raddbgi_breakpad_from_pdb_main.c            %compile_link% %out%raddbgi_breakpad_from_pdb.exe || exit /b 1
+if "%ryan_scratch%"=="1"               %compile%             ..\src\scratch\ryan_scratch.c                                                %compile_link% %out%ryan_scratch.exe || exit /b 1
+if "%cpp_tests%"=="1"                  %compile%             ..\src\scratch\i_hate_c_plus_plus.cpp                                        %compile_link% %out%cpp_tests.exe || exit /b 1
+if "%look_at_raddbg%"=="1"             %compile%             ..\src\scratch\look_at_raddbg.c                                              %compile_link% %out%look_at_raddbg.exe || exit /b 1
+if "%mule_main%"=="1"                  del vc*.pdb mule*.pdb && %compile_release% %only_compile% ..\src\mule\mule_inline.cpp && %compile_release% %only_compile% ..\src\mule\mule_o2.cpp && %compile_debug% %EHsc% ..\src\mule\mule_main.cpp ..\src\mule\mule_c.c mule_inline.obj mule_o2.obj %compile_link% %out%mule_main.exe || exit /b 1
+if "%mule_module%"=="1"                %compile%             ..\src\mule\mule_module.cpp                                                  %compile_link% %link_dll% %out%mule_module.dll || exit /b 1
+if "%mule_hotload%"=="1"               %compile% ..\src\mule\mule_hotload_main.c %compile_link% %out%mule_hotload.exe & %compile% ..\src\mule\mule_hotload_module_main.c %compile_link% %link_dll% %out%mule_hotload_module.dll || exit /b 1
 popd
 
 :: --- Unset ------------------------------------------------------------------