Introduce module and debug info finder APIs

drgn currently provides limited control over how debugging information is found. drgn has hardcoded logic for where to search for debugging information. The most the user can do is provide a list of files for drgn to try in addition to the default locations (with the -s CLI option or the drgn.Program.load_debug_info() method). The implementation is also a mess. We use libdwfl, but its data model is slightly different from what we want, so we have to work around it or reimplement its functionality in several places: see commits e5874ad ("libdrgn: use libdwfl"), e6abfea ("libdrgn: debug_info: report userspace core dump debug info ourselves"), and 1d4854a ("libdrgn: implement optimized x86-64 ELF relocations") for some examples. The mismatched combination of libdwfl and our own code is difficult to maintain, and the lack of control over the whole debug info pipeline has made it difficult to fix several longstanding issues. The solution is a major rework removing our libdwfl dependency and replacing it with our own model. This (huge) commit is that rework comprising the following components: - drgn.Module/struct drgn_module, a representation of a binary used by a program. - Automatic discovery of the modules loaded in a program. - Interfaces for manually creating and overriding modules. - Automatic discovery of debugging information from the standard locations and debuginfod. - Interfaces for custom debug info finders and for manually overriding debugging information. - Tons of test cases. A lot of care was taken to make these interfaces extremely flexible yet cohesive. The existing interfaces are also reimplemented on top of the new functionality to maintain backwards compatibility, with one exception: drgn.Program.load_debug_info()/-s would previously accept files that it didn't find loaded in the program. This turned out to be a big footgun for users, so now this must be done explicitly (with drgn.ExtraModule/--extra-symbols). The API and implementation both owe a lot to libdwfl: - The concepts of modules, module address ranges/section addresses, and file biases are heavily inspired by the libdwfl interfaces. - Ideas for determining modules in userspace processes and core dumps were taken from libdwfl. - Our implementation of ELF symbol table address lookups is based on dwfl_module_addrinfo(). drgn has taken these concepts and fine-tuned them based on lessons learned. Credit is also due to Stephen Brennan for early testing and feedback. Closes #16, closes #25, closes #332. Signed-off-by: Omar Sandoval <osandov@osandov.com>
osandov · Dec 18, 2024 · cbf2572 · cbf2572
1 parent fce038c
commit cbf2572
Show file tree

Hide file tree

Showing 61 changed files with 14,074 additions and 3,641 deletions.
diff --git a/_drgn.pyi b/_drgn.pyi
diff --git a/docs/advanced_usage.rst b/docs/advanced_usage.rst
@@ -6,15 +6,128 @@ Advanced Usage
 The :doc:`user_guide` covers basic usage of drgn, but drgn also supports more
 advanced use cases which are covered here.
 
-Loading Debugging Symbols
--------------------------
+.. _advanced-modules:
 
-drgn will automatically load debugging information based on the debugged
-program (e.g., from loaded kernel modules or loaded shared libraries).
-:meth:`drgn.Program.load_debug_info()` can be used to load additional debugging
-information::
+Modules and Debugging Symbols
+-----------------------------
 
-    >>> prog.load_debug_info(['./libfoo.so', '/usr/lib/libbar.so'])
+drgn tries to determine what executable, libraries, etc. a program uses and
+load debugging symbols automatically. As long as :doc:`debugging symbols are
+installed <getting_debugging_symbols>`, this should work out of the box on
+standard setups.
+
+For non-standard scenarios, drgn allows overriding the defaults with different
+levels of control and complexity.
+
+Loading Debugging Symbols From Non-Standard Locations
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+drgn searches standard locations for debugging symbols. If you have debugging
+symbols available in a non-standard location, you can provide it to the CLI
+with the ``-s``/``--symbols`` option:
+
+.. code-block:: console
+
+    $ drgn -s ./libfoo.so -s /usr/lib/libbar.so.debug
+
+Or with the :meth:`drgn.Program.load_debug_info()` method::
+
+    >>> prog.load_debug_info(["./libfoo.so", "/usr/lib/libbar.so.debug"])
+
+Loading Debugging Symbols For Specific Modules
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+``-s`` and ``load_debug_info()`` try the given files against all of the modules
+loaded in the program based on build IDs. You can also :ref:`look up
+<api-module-constructors>` a specific module and try a given file for just that
+module with :meth:`drgn.Module.try_file()`::
+
+    >>> prog.main_module().try_file("build/vmlinux")
+
+Loading Additional Debugging Symbols
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+``-s`` and ``load_debug_info()`` ignore files that don't correspond to a loaded
+module. To load debugging symbols from an arbitrary file, pass
+``--extra-symbols`` to the CLI:
+
+.. code-block:: console
+
+    $ drgn --extra-symbols ./my_extra_symbols.debug
+
+Or create a :class:`drgn.ExtraModule`::
+
+    >>> module = prog.extra_module("my_extra_symbols")
+    >>> module.try_file("./my_extra_symbols.debug")
+
+Listing Modules
+^^^^^^^^^^^^^^^
+
+By default, drgn creates a module for everything loaded in the program. You can
+disable this in the CLI with ``-no-default-symbols``.
+
+You can find or create the loaded modules programmatically with
+:meth:`drgn.Program.loaded_modules()`::
+
+    >>> for module, new in prog.loaded_modules():
+    ...     print("Created" if new else "Found", module)
+
+You can see all of the created modules with :meth:`drgn.Program.modules()`.
+
+Overriding Modules
+^^^^^^^^^^^^^^^^^^
+
+You can create modules with the :ref:`module factory functions
+<api-module-constructors>`. You can also modify various attributes of the
+:class:`drgn.Module` class.
+
+Debug Info Finders
+^^^^^^^^^^^^^^^^^^
+
+A callback for automatically finding debugging symbols for a set of modules can
+be registered with :meth:`drgn.Program.register_debug_info_finder()`. Here is
+an example for getting debugging symbols on Fedora Linux using DNF:
+
+.. code-block:: python3
+
+    import subprocess
+
+    import drgn
+
+    # Install debugging symbols using the DNF debuginfo-install plugin. Note that
+    # this is mainly for demonstration purposes; debuginfod, which drgn supports
+    # out of the box, is more reliable.
+    def dnf_debug_info_finder(modules: list[drgn.Module]) -> None:
+        packages = set()
+        for module in modules:
+            if not module.wants_debug_file():
+                continue
+
+            if not module.name.startswith("/"):
+                continue
+
+            proc = subprocess.run(
+                ["rpm", "--query", "--file", module.name],
+                stdout=subprocess.PIPE,
+                stderr=subprocess.DEVNULL,
+                text=True,
+            )
+            if proc.returncode == 0:
+                packages.add(proc.stdout.rstrip("\n"))
+
+        # Try installing their debug info.
+        subprocess.call(
+            ["sudo", "dnf", "debuginfo-install", "--skip-broken", "--"]
+            + sorted(packages)
+        )
+
+        # Leave the rest to the standard debug info finder.
+
+
+    prog.register_debug_info_finder("dnf", dnf_debug_info_finder, enable_index=0)
+
+Currently, debug info finders must be configured explicitly by the user. In the
+future, there will be a plugin system for doing so automatically.
 
 Library
 -------
@@ -92,9 +205,9 @@ Environment Variables
 Some of drgn's behavior can be modified through environment variables:
 
 ``DRGN_MAX_DEBUG_INFO_ERRORS``
-    The maximum number of individual errors to report in a
-    :exc:`drgn.MissingDebugInfoError`. Any additional errors are truncated. The
-    default is 5; -1 is unlimited.
+    The maximum number of warnings about missing debugging information to log
+    on CLI startup or from :meth:`drgn.Program.load_debug_info()`. Any
+    additional errors are truncated. The default is 5; -1 is unlimited.
 
 ``DRGN_PREFER_ORC_UNWINDER``
     Whether to prefer using `ORC
@@ -104,12 +217,6 @@ Some of drgn's behavior can be modified through environment variables:
     vice versa. This environment variable is mainly intended for testing and
     may be ignored in the future.
 
-``DRGN_USE_LIBDWFL_REPORT``
-    Whether drgn should use libdwfl to find debugging information for core
-    dumps instead of its own implementation (0 or 1). The default is 0. This
-    environment variable is mainly intended as an escape hatch in case of bugs
-    in drgn's implementation and will be ignored in the future.
-
 ``DRGN_USE_LIBKDUMPFILE_FOR_ELF``
     Whether drgn should use libkdumpfile for ELF vmcores (0 or 1). The default
     is 0. This functionality will be removed in the future.

diff --git a/docs/api_reference.rst b/docs/api_reference.rst
@@ -7,7 +7,7 @@ Programs
 --------
 
 .. drgndoc:: Program
-    :exclude: (void|int|bool|float|struct|union|class|enum|typedef|pointer|array|function)_type
+    :exclude: (void|int|bool|float|struct|union|class|enum|typedef|pointer|array|function)_type|(main|shared_library|vdso|relocatable|linux_kernel_loadable|extra)_module
 .. drgndoc:: ProgramFlags
 .. drgndoc:: FindObjectFlags
 
@@ -159,6 +159,44 @@ can be used just like types obtained from :meth:`Program.type()`.
 .. drgndoc:: Program.array_type
 .. drgndoc:: Program.function_type
 
+Modules
+-------
+
+.. drgndoc:: Module
+.. drgndoc:: MainModule
+.. drgndoc:: SharedLibraryModule
+.. drgndoc:: VdsoModule
+.. drgndoc:: RelocatableModule
+.. drgndoc:: ExtraModule
+.. drgndoc:: ModuleFileStatus
+.. drgndoc:: WantedSupplementaryFile
+.. drgndoc:: SupplementaryFileKind
+
+.. _api-module-constructors:
+
+Module Lookups/Constructors
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+For each module type, there is a corresponding method to create a module of
+that type or find one that was previously created::
+
+    >>> prog.extra_module("foo", 1234)
+    Traceback (most recent call last):
+      ...
+    LookupError: module not found
+    >>> prog.extra_module("foo", 1234, create=True)
+    (prog.extra_module(name='foo', id=0x4d2), True)
+    >>> prog.extra_module("foo", 1234)
+    >>> prog.extra_module("foo", 1234, create=True)
+    (prog.extra_module(name='foo', id=0x4d2), False)
+
+.. drgndoc:: Program.main_module
+.. drgndoc:: Program.shared_library_module
+.. drgndoc:: Program.vdso_module
+.. drgndoc:: Program.relocatable_module
+.. drgndoc:: Program.linux_kernel_loadable_module
+.. drgndoc:: Program.extra_module
+
 Miscellaneous
 -------------
 

diff --git a/docs/user_guide.rst b/docs/user_guide.rst
@@ -319,6 +319,47 @@ functions like :meth:`drgn.Program.int_type()`::
 You won't usually need to work with types directly, but see
 :ref:`api-reference-types` if you do.
 
+Modules
+^^^^^^^
+
+drgn tracks executables, shared libraries, loadable kernel modules, and other
+binary files used by a program with the :class:`drgn.Module` class. Modules
+store their name, identifying information, load address, and debugging symbols.
+
+.. code-block:: pycon
+    :caption: Linux kernel example
+
+    >>> for module in prog.modules():
+    ...     print(module)
+    ...
+    prog.main_module(name='kernel')
+    prog.relocatable_module(name='rng_core', address=0xffffffffc0400000)
+    prog.relocatable_module(name='virtio_rng', address=0xffffffffc0402000)
+    prog.relocatable_module(name='binfmt_misc', address=0xffffffffc0401000)
+    >>> prog.main_module().debug_file_path
+    '/usr/lib/modules/6.13.0-rc1-vmtest34.1default/build/vmlinux'
+
+.. code-block:: pycon
+    :caption: Userspace example
+
+    >>> for module in prog.modules():
+    ...     print(module)
+    ...
+    prog.main_module(name='/usr/bin/grep')
+    prog.shared_library_module(name='/lib64/ld-linux-x86-64.so.2', dynamic_address=0x7f51772b6e68)
+    prog.shared_library_module(name='/lib64/libc.so.6', dynamic_address=0x7f51771af960)
+    prog.shared_library_module(name='/lib64/libpcre2-8.so.0', dynamic_address=0x7f5177258c68)
+    prog.vdso_module(name='linux-vdso.so.1', dynamic_address=0x7f51772803e0)
+    >>> prog.main_module().loaded_file_path
+    '/usr/bin/grep'
+    >>> prog.main_module().debug_file_path
+    '/usr/lib/debug/usr/bin/grep-3.11-7.fc40.x86_64.debug'
+
+drgn normally initializes the appropriate modules and loads their debugging
+symbols automatically. Advanced use cases can create or modify modules and load
+debugging symbols manually; see the :ref:`advanced usage guide
+<advanced-modules>`.
+
 Platforms
 ^^^^^^^^^
 

diff --git a/drgn/__init__.py b/drgn/__init__.py
@@ -49,11 +49,15 @@
 from _drgn import (
     NULL,
     Architecture,
+    ExtraModule,
     FaultError,
     FindObjectFlags,
     IntegerLike,
     Language,
+    MainModule,
     MissingDebugInfoError,
+    Module,
+    ModuleFileStatus,
     NoDefaultProgramError,
     Object,
     ObjectAbsentError,
@@ -66,8 +70,11 @@
     ProgramFlags,
     Qualifiers,
     Register,
+    RelocatableModule,
+    SharedLibraryModule,
     StackFrame,
     StackTrace,
+    SupplementaryFileKind,
     Symbol,
     SymbolBinding,
     SymbolIndex,
@@ -80,6 +87,8 @@
     TypeMember,
     TypeParameter,
     TypeTemplateParameter,
+    VdsoModule,
+    WantedSupplementaryFile,
     alignof,
     cast,
     container_of,
@@ -106,11 +115,15 @@
 
 __all__ = (
     "Architecture",
+    "ExtraModule",
     "FaultError",
     "FindObjectFlags",
     "IntegerLike",
     "Language",
+    "MainModule",
     "MissingDebugInfoError",
+    "Module",
+    "ModuleFileStatus",
     "NULL",
     "NoDefaultProgramError",
     "Object",
@@ -124,8 +137,11 @@
     "ProgramFlags",
     "Qualifiers",
     "Register",
+    "RelocatableModule",
+    "SharedLibraryModule",
     "StackFrame",
     "StackTrace",
+    "SupplementaryFileKind",
     "Symbol",
     "SymbolBinding",
     "SymbolIndex",
@@ -138,6 +154,8 @@
     "TypeMember",
     "TypeParameter",
     "TypeTemplateParameter",
+    "VdsoModule",
+    "WantedSupplementaryFile",
     "alignof",
     "cast",
     "container_of",