Module addresses loading as relative rather than absolute

[brenns10]

Hi Omar, this may be because of incomplete testing I did back on #178... but I'm not certain. I had a team member report that after loading a kernel module's debuginfo, module symbols became relative to their load address, rather than absolute. Here's what I mean:

$ drgn -c vmcore -s vmlinux -s 5.4.17-2136.305.3.el8uek.local.x86_64/virtio_console.ko.debug
drgn 0.0.20 (using Python 3.6.8, elfutils 0.187, with libkdumpfile)
could not get debugging information for:
kernel modules (could not read depmod: open: /lib/modules/5.4.17-2136.305.3.el8uek.local.x86_64/modules.dep.bin: No such file or directory)
For help, type help(drgn).
>>> import drgn
>>> from drgn import NULL, Object, cast, container_of, execscript, offsetof, reinterpret, sizeof
>>> from drgn.helpers.linux import *
>>> prog.symbol("virtio_console")
Symbol(name='virtio_console', address=0x120, size=0x108, binding=<SymbolBinding.LOCAL: 1>, kind=<SymbolKind.OBJECT: 1>)

# Compare to:
$ drgn -c vmcore -s vmlinux -s 5.4.17-2136.305.3.el8uek.local.x86_64/virtio_console.ko.debug
drgn 0.0.19 (using Python 3.6.8, elfutils 0.186, with libkdumpfile)
could not get debugging information for:
kernel modules (could not read depmod: open: /lib/modules/5.4.17-2136.305.3.el8uek.local.x86_64/modules.dep.bin: No such file or directory)
For help, type help(drgn).
>>> import drgn
>>> from drgn import NULL, Object, cast, container_of, execscript, offsetof, reinterpret, sizeof
>>> from drgn.helpers.linux import *
>>> prog.symbol("virtio_console")
Symbol(name='virtio_console', address=0xffffffffc034d120, size=0x108, binding=<SymbolBinding.LOCAL: 1>, kind=<SymbolKind.OBJECT: 1>)

I also am reminded of #185... I tried out your module-percpu-hack branch and it also gives me the address=0x120 value.

[osandov]

Huh, so that suggests that 0.0.19 was succeeding in matching the module by name, but 0.0.20 is failing to match by build ID. Could you please provide the output of running your reproducer with this patch:

diff --git a/libdrgn/linux_kernel.c b/libdrgn/linux_kernel.c
index 90c2aa8..e921c6a 100644
--- a/libdrgn/linux_kernel.c
+++ b/libdrgn/linux_kernel.c
@@ -1309,6 +1309,11 @@ report_loaded_kernel_module(struct drgn_debug_info_load_state *load,
 						    err);
 	}
 
+	fprintf(stderr, "module %s has build ID ", kmod_it->name);
+	for (size_t i = 0; i < key.len; i++)
+		fprintf(stderr, "%02" PRIx8, ((uint8_t *)key.str)[i]);
+	fputc('\n', stderr);
+
 	struct hash_pair hp = kernel_module_table_hash(&key);
 	struct kernel_module_table_iterator it =
 		kernel_module_table_search_hashed(kmod_table, &key, hp);

and the output of running readelf -S 5.4.17-2136.305.3.el8uek.local.x86_64/virtio_console.ko.debug and readelf -n 5.4.17-2136.305.3.el8uek.local.x86_64/virtio_console.ko.debug?

[brenns10]

Ok, here we go:

stepbren at stepbren-lnx in ~/Temporary/2022-08-10_module-drgn-issue
$ readelf -S 5.4.17-2136.305.3.el8uek.local.x86_64/virtio_console.ko.debug
There are 55 section headers, starting at offset 0xac070:

Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
  [ 0]                   NULL             0000000000000000  00000000
       0000000000000000  0000000000000000           0     0     0
  [ 1] .note.gnu.bu[...] NOTE             0000000000000000  00000040
       0000000000000024  0000000000000000   A       0     0     4
  [ 2] .note.Linux       NOTE             0000000000000000  00000064
       0000000000000018  0000000000000000   A       0     0     4
  [ 3] .text             PROGBITS         0000000000000000  00000080
       0000000000003307  0000000000000000  AX       0     0     16
  [ 4] .rela.text        RELA             0000000000000000  0005c790
       0000000000002c70  0000000000000018   I      52     3     8
  [ 5] .text..refcount   PROGBITS         0000000000000000  00003387
       0000000000000020  0000000000000000  AX       0     0     1
  [ 6] .init.text        PROGBITS         0000000000000000  000033a7
       0000000000000129  0000000000000000  AX       0     0     1
  [ 7] .rela.init.text   RELA             0000000000000000  0005f400
       0000000000000378  0000000000000018   I      52     6     8
  [ 8] .exit.text        PROGBITS         0000000000000000  000034d0
       000000000000003b  0000000000000000  AX       0     0     1
  [ 9] .rela.exit.text   RELA             0000000000000000  0005f778
       00000000000000d8  0000000000000018   I      52     8     8
  [10] __mcount_loc      PROGBITS         0000000000000000  0000350b
       00000000000001c8  0000000000000000   A       0     0     1
  [11] .rela__mcount_loc RELA             0000000000000000  0005f850
       0000000000000558  0000000000000018   I      52    10     8
  [12] .rodata.str1.1    PROGBITS         0000000000000000  000036d3
       0000000000000236  0000000000000001 AMS       0     0     1
  [13] .smp_locks        PROGBITS         0000000000000000  0000390c
       0000000000000018  0000000000000000   A       0     0     4
  [14] .rela.smp_locks   RELA             0000000000000000  0005fda8
       0000000000000090  0000000000000018   I      52    13     8
  [15] .rodata.str1.8    PROGBITS         0000000000000000  00003928
       00000000000002a3  0000000000000001 AMS       0     0     8
  [16] .parainstructions PROGBITS         0000000000000000  00003bd0
       000000000000028c  0000000000000000   A       0     0     8
  [17] .rela.parain[...] RELA             0000000000000000  0005fe38
       00000000000003d8  0000000000000018   I      52    16     8
  [18] __ex_table        PROGBITS         0000000000000000  00003e5c
       0000000000000030  0000000000000000   A       0     0     4
  [19] .rela__ex_table   RELA             0000000000000000  00060210
       0000000000000120  0000000000000018   I      52    18     8
  [20] .rodata           PROGBITS         0000000000000000  00003ea0
       00000000000003e0  0000000000000000   A       0     0     32
  [21] .rela.rodata      RELA             0000000000000000  00060330
       00000000000001f8  0000000000000018   I      52    20     8
  [22] .modinfo          PROGBITS         0000000000000000  00004280
       0000000000000100  0000000000000000   A       0     0     1
  [23] __versions        PROGBITS         0000000000000000  00004380
       0000000000001980  0000000000000000   A       0     0     32
  [24] __jump_table      PROGBITS         0000000000000000  00005d00
       0000000000000030  0000000000000000  WA       0     0     8
  [25] .rela__jump_table RELA             0000000000000000  00060528
       00000000000000d8  0000000000000018   I      52    24     8
  [26] .data             PROGBITS         0000000000000000  00005d40
       0000000000000318  0000000000000000  WA       0     0     32
  [27] .rela.data        RELA             0000000000000000  00060600
       0000000000000210  0000000000000018   I      52    26     8
  [28] __bug_table       PROGBITS         0000000000000000  00006058
       0000000000000048  0000000000000000  WA       0     0     1
  [29] .rela__bug_table  RELA             0000000000000000  00060810
       0000000000000120  0000000000000018   I      52    28     8
  [30] __verbose         PROGBITS         0000000000000000  000060a0
       0000000000000070  0000000000000000  WA       0     0     8
  [31] .rela__verbose    RELA             0000000000000000  00060930
       00000000000000c0  0000000000000018   I      52    30     8
  [32] .gnu.linkonc[...] PROGBITS         0000000000000000  00006140
       0000000000000380  0000000000000000  WA       0     0     64
  [33] .rela.gnu.li[...] RELA             0000000000000000  000609f0
       0000000000000030  0000000000000018   I      52    32     8
  [34] .bss              NOBITS           0000000000000000  000064c0
       0000000000000014  0000000000000000  WA       0     0     8
  [35] .comment          PROGBITS         0000000000000000  000064c0
       0000000000000062  0000000000000001  MS       0     0     1
  [36] .note.GNU-stack   PROGBITS         0000000000000000  00006522
       0000000000000000  0000000000000000           0     0     1
  [37] .debug_info       PROGBITS         0000000000000000  00006522
       0000000000026db4  0000000000000000           0     0     1
  [38] .rela.debug_info  RELA             0000000000000000  00060a20
       0000000000038238  0000000000000018   I      52    37     8
  [39] .debug_abbrev     PROGBITS         0000000000000000  0002d2d6
       00000000000011b3  0000000000000000           0     0     1
  [40] .debug_loc        PROGBITS         0000000000000000  0002e489
       0000000000003dd0  0000000000000000           0     0     1
  [41] .rela.debug_loc   RELA             0000000000000000  00098c58
       0000000000006bd0  0000000000000018   I      52    40     8
  [42] .debug_aranges    PROGBITS         0000000000000000  00032259
       0000000000000080  0000000000000000           0     0     1
  [43] .rela.debug_[...] RELA             0000000000000000  0009f828
       0000000000000090  0000000000000018   I      52    42     8
  [44] .debug_ranges     PROGBITS         0000000000000000  000322d9
       0000000000004f30  0000000000000000           0     0     1
  [45] .rela.debug_[...] RELA             0000000000000000  0009f8b8
       000000000000b8e0  0000000000000018   I      52    44     8
  [46] .debug_line       PROGBITS         0000000000000000  00037209
       0000000000005c76  0000000000000000           0     0     1
  [47] .rela.debug_line  RELA             0000000000000000  000ab198
       00000000000000a8  0000000000000018   I      52    46     8
  [48] .debug_str        PROGBITS         0000000000000000  0003ce7f
       00000000000136a1  0000000000000001  MS       0     0     1
  [49] .debug_frame      PROGBITS         0000000000000000  00050520
       0000000000000e80  0000000000000000           0     0     8
  [50] .rela.debug_frame RELA             0000000000000000  000ab240
       0000000000000c30  0000000000000018   I      52    49     8
  [51] .BTF              PROGBITS         0000000000000000  000513a0
       0000000000008f48  0000000000000000           0     0     1
  [52] .symtab           SYMTAB           0000000000000000  0005a2e8
       0000000000001728  0000000000000018          53   140     8
  [53] .strtab           STRTAB           0000000000000000  0005ba10
       0000000000000d7e  0000000000000000           0     0     1
  [54] .shstrtab         STRTAB           0000000000000000  000abe70
       0000000000000200  0000000000000000           0     0     1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  D (mbind), l (large), p (processor specific)

stepbren at stepbren-lnx in ~/Temporary/2022-08-10_module-drgn-issue
$ readelf -n 5.4.17-2136.305.3.el8uek.local.x86_64/virtio_console.ko.debug

Displaying notes found in: .note.gnu.build-id
  Owner                Data size        Description
  GNU                  0x00000014       NT_GNU_BUILD_ID (unique build ID bitstring)
    Build ID: 0da385e445bc4ef3fc1f90ddacf0e2ea91416f07

Displaying notes found in: .note.Linux
  Owner                Data size        Description
  Linux                0x00000001       OPEN
   description data: 00

And for the drgn output with the patch:

$ python -m drgn -c /home/stepbren/Temporary/2022-08-10_module-drgn-issue/vmcore -s /home/stepbren/Temporary/2022-08-10_module-drgn-issue/vmlinux -s /home/stepbren/Temporary/2022-08-10_module-drgn-issue/5.4.17-2136.305.3.el8uek.local.x86_64/virtio_console.ko.debug
drgn 0.0.20+6.g4296653.dirty (using Python 3.10.4, elfutils 0.186, with libkdumpfile)
module scsi_transport_iscsi has build ID fe05c2d002fe2ecd98bd319152789016195385d5
module nft_fib_inet has build ID 90cd3589f78f61c2045c12466fd9ca3673cf4dd8
module nft_fib_ipv4 has build ID 7ea32c1fb43ae3d916896ef50e076383d5f56f92
module nft_fib_ipv6 has build ID 0f420db40d95b458dc0b69f22d7db2d5dcf46148
module nft_fib has build ID 45de8b25f8f3b84d02ffdaf75dff10f10e151a5d
module nft_reject_inet has build ID 82ebe09543b139999ae6ef5afe41da4ecae726d3
module nf_reject_ipv4 has build ID 02c484ee1d8787f1d7306a5f6a0ca8596046c978
module nf_reject_ipv6 has build ID 42a8a641a0340427fc6b367a98ab188c1077e641
module nft_reject has build ID d7c8964ce7c3ae3482816e85ee49da79e1b75320
module nft_ct has build ID eaa94f071de725cfba06f258c49c33c48b5d4615
module nf_tables_set has build ID 701e92cb5de2298b562fe5fe501d9d0f1c32f38d
module nft_chain_nat has build ID 50894da056b168a3b900949853331bb341af65ea
module nf_nat has build ID c01ebbf194879be8dbad6fd7f84ed5c104a74521
module nf_conntrack has build ID bb50b93ef32a7ddccd0deaa16ce1903239cff434
module nf_defrag_ipv6 has build ID 69d28307c158781b2e91aec438ea27a33d6f80ea
module nf_defrag_ipv4 has build ID 6a4472d1dbb9657fa084014801f4e59f4ae70440
module ip_set has build ID 918a113de7bc45ce7c203ddc0d31923b2a080e3d
module rfkill has build ID 180a10e38abd59cb033f059e8388f6d945a1dddf
module sunrpc has build ID 6ed7c38648283c63487ef88341d82c800486af44
module intel_rapl_msr has build ID d3abfe9c2c8e2407c529acd599c10d3e2f533447
module intel_rapl_common has build ID 80d00235bbac3238ff63908d277a78c5b18b41b6
module crct10dif_pclmul has build ID 696bab84a83ee56722e379fd8f4b747cfb78aed9
module crc32_pclmul has build ID 74aa6dce263a94d4c84b2d359e9991b06698e862
module ghash_clmulni_intel has build ID 655e972337b878f00c321ebe6c27670cc65e4df8
module aesni_intel has build ID cf5f878dcd2956bb077d9bf1a63a9cf6d7df2e73
module crypto_simd has build ID d2ad17916ff573500d65ea6de0834c16733ae05e
module cryptd has build ID 234e0ed99800dffabb454ee99f78ef192aec21ba
module glue_helper has build ID e2751e03475bdbc44a05ec23da816412808f3623
module i2c_piix4 has build ID 4f876df029fbb735e5255e54b7b8c225fb23bfbc
module virtio_balloon has build ID 7e02a4b32758cc1f4ab2ebb0ffdc30b5c057061f
module pcspkr has build ID 48a075e22460e0164ce3036718044cc52ccaecc1
module joydev has build ID 6e08b29019252a61f69d85c509591cbe7f944a97
module sch_fq_codel has build ID c2a7e892cfa9f83018e5d8dd0cbabab67f11999d
module xfs has build ID 3f70e52a1e9a0d1330ec8f20afda9087f90f5885
module libcrc32c has build ID 7eaeaa721c89e9ec9fcde9efe19b79e90eb17a01
module sr_mod has build ID b599e15f09da0ad33b191586f9ab1593751a53b1
module cdrom has build ID f0efa2e946ed84babec17cd4b17d319c8f0b3535
module sd_mod has build ID 6cbec15b16231ad4ae894e6f9d0d8f32115d8de2
module sg has build ID 9b429a178425cf7c4356aee18cfe34dca5affb1d
module ata_generic has build ID bed83e7c8cc9d553f3ff21f1e9c6ba81d4f5c000
module pata_acpi has build ID 5be2de36d7b3981dd4da227dad08203feb701892
module virtio_net has build ID 7e8530e667d15d02e82aac829c23e84f4b55b10f
module net_failover has build ID a4c9503a1527460cca18fddc6b8b1803c9e995c4
module virtio_scsi has build ID 1e69d992f31bae863c68974056edf60ffad2482c
module virtio_console has build ID fe343115835a454a2e1037b98afba60842c5b1ea
module failover has build ID 1b3c136faea82adf789090391531973a1c387dc6
module bochs_drm has build ID f3dff3aa41a90ec4f7ace356ce520dcb03359e03
module drm_kms_helper has build ID 4700f382f7b9e74167e0bc7aff6ef32a31ab7cfa
module syscopyarea has build ID 0e1108915ff55342f1686155830f75ab3370af4f
module sysfillrect has build ID d2156aeb9ae84f5e3da0ed4b881d5c291909202b
module sysimgblt has build ID 1aaf3913972fac39c7e327d811136a1d4ff99d78
module fb_sys_fops has build ID 618b571627a59d94765e795c417e70225aa3eadd
module drm_vram_helper has build ID 4a2a62370e00b02a83cec43164865d26ff10780f
module ttm has build ID ff46eff6f2164cf8e052d3816e093ad8237a02ce
module drm has build ID 3370fc574745cb71583c8f8a0678764d109a7d17
module serio_raw has build ID 4eae7aa2a174ba4dff47de75137574d814222e91
module floppy has build ID 5e6654d2109558d7fc7202f74804f05767543a43
module ata_piix has build ID 3c8c42840a69c0663c3190b55009734da171fc7a
module libata has build ID be6bc8a44d9c58e218ca69900aa2a517ad33213e
module virtio_pci has build ID 9f6489df410661fdecbb427826c551a46e5e9502
module virtio_pci_modern_dev has build ID 13494f4be085d8cc0e3e5faa4949e0cbd4adaab9
module qemu_fw_cfg has build ID 30b2526b61397d46041d4a93f8c7f3f068fde318
module dm_mirror has build ID b7df6449863f56ca805010240568c7930f560820
module dm_region_hash has build ID 10d395c07d98ba97a4fd06a7a65576241f4cb402
module dm_log has build ID 01a9031c8f46bc8779f8e9f1a5633beb638b7cd1
module dm_mod has build ID d7a0a3250b0cca6a686b66c133c47033dfa5b2b3
module fuse has build ID 2010f4f9a1a4caf2bfd4c02f8620250cce8358be
module virtio_rng has build ID 7b31d50b0a2a3d4ccf2f3b41dfb41026b39ea6d2
could not get debugging information for:
kernel modules (could not read depmod: open: /lib/modules/5.4.17-2136.305.3.el8uek.local.x86_64/modules.dep.bin: No such file or directory)
For help, type help(drgn).
>>> import drgn
>>> from drgn import NULL, Object, cast, container_of, execscript, offsetof, reinterpret, sizeof
>>> from drgn.helpers.linux import *
>>> prog["init_uts"].ns

[brenns10]

Clearly they don't match :/
I guess this may be a build-side error from our kernel? I can't imagine how these build-id's don't match though.

[osandov]

Do you have the loadable .ko file available as well? It'd also be useful to see the output of readelf -n on that file, to see whether that build ID matches the .ko.debug file, the build ID that drgn parsed out of the vmcore, or neither.

[brenns10]

I am chasing after the .ko file for this build. It looks like it was a local build, so I think it could be possible that an older version got preserved or overwritten. I got the .ko and .ko.debug for a production build on a similar version, and I see that the Build ID's match there, so there's no systematic build ID mismatch in our system. This may be user error rather than a bug. I'll update when I have both files.

I do wonder though, is this the behavior when a mismatched build ID is loaded? Is it just loaded as if it were another vmlinux debuginfo? I wonder if it would be better to just raise an error, skip, or at least require a "force=True" argument to load a mismatched build ID? I can see somebody saying "i know what I'm doing" and being right, but in the general case, it seems like a very confusing error case. I'd rather get an exception during loading, than get a weird error down the line.

Alternatively, if there were a way to view the loaded debuginfo files, and see which ones were matched to which modules, then I'd imagine we could introspect this a bit better. Just spitballing ideas though, I'm really unfamiliar with this part of the code...

[osandov]

Currently, "leftover" files given to -s/load_debug_info() that don't match any loaded kernel modules are added with no relocations or adjustments. The idea behind that was to allow looking at types and such in arbitrary files without needing them loaded in a process or core dump. E.g., drgn -p0 -s whatever.so runs drgn against itself and lets you look up stuff from whatever.so (-p0 is an undocumented feature that means "debug drgn's own PID").

But you're right, when this isn't intentional, it causes this really surprising breakage. The debug info discovery rework I'm working on should help address this. Essentially, I'm adding APIs for asking drgn "what modules do you think are loaded in this core dump?", then for each module, specifying what files to use for each module, and fabricating modules for cases where drgn either gets it wrong or you want to do something weird like my whatever.so use case above. It also adds more logging (clearly that would've been useful here). Once that's in place, I think it'd be a good idea to make -s stricter (only use it for modules that we can specifically match to) and either have a new command line option for my whatever.so use case or just use the new Python API. It'll be glorious once I get back to finishing it 🥲

[brenns10]

Awesome! I wish there was something I could do to help with the debuginfo discovery rework, it sounds great. Will there be a possibility to specify another search path for debuginfo? For example, we frequently extract RPMs, or at least the contents of /usr/lib/debug/lib/modules/$(uname -r)/, to a shared location on our vmcore analysis servers. We can specify debuginfo files manually in our wrapper which is no problem (and I bet the newer Python API will make this even better). But if drgn knew another search path, then it could even be automatic.

Still waiting on that .ko file, but I'd really be shocked if there was anything other than a mismatch here.

[brenns10]

For what it's worth, a gross quicky for getting the build ID without patching drgn :)

def get_buildid(module):
    for i in range(module.notes_attrs.notes):
        if module.notes_attrs.attrs[i].attr.name.string_() == b".note.gnu.build-id":
            return module.prog_.read(module.notes_attrs.attrs[i].private.value_(), 36)[-20:].hex()

[brenns10]

The .ko seems to be unrecoverable at this point. I'm going to close the issue because I haven't observed a build ID mismatch in our production kernels and, if there were, that would be our bug, not drgn's :)

[osandov]

Awesome! I wish there was something I could do to help with the debuginfo discovery rework, it sounds great. Will there be a possibility to specify another search path for debuginfo? For example, we frequently extract RPMs, or at least the contents of /usr/lib/debug/lib/modules/$(uname -r)/, to a shared location on our vmcore analysis servers. We can specify debuginfo files manually in our wrapper which is no problem (and I bet the newer Python API will make this even better). But if drgn knew another search path, then it could even be automatic.

I think @sdimitro asked for this before and I opened #17 for it, so I'll definitely take that into account. I don't imagine that /usr/lib/debug/lib/modules/$(uname -r)/ has a modules.dep.bin index, right? drgn uses that to map a module name to the .ko path. Without it, I think we'll need to do a walk over the whole modules directory to find the modules we need by filename.

[brenns10]

Correct, there's no modules.dep.bin in /usr/lib/debug/lib/modules/$(uname -r)/. However, I did just check, and drgn (0.0.20) still finds the debuginfo without that file. Maybe you already implemented the walk and search?

[osandov]

Oh, in that case, it's probably finding it by build ID under /usr/lib/debug/.build-id. It'll be really easy to support an alternative /usr/lib/debug directory.

[brenns10]

Wow, I had no idea that /usr/lib/debug/.build-id/ existed. Who designed "hidden" dotfiles anyway? Somewhat frustrating... 😝

That said, while our "-debuginfo" packages install the .build-id directory, our analysis systems don't have that file. On our vmcore analysis systems, we simply extract the files out of the package and put them into a directory organized like this:

$root_path/$num_bits/$(uname -r)/vmlinux

eg:
$root_path/64/5.4.17-2136.309.5.el7uek.x86_64/vmlinux

And I think modules get placed alongside vmlinux. I guess we could make some scripts to maintain a .build-id directory ourselves, I think it would just be a few lines of wrangling readelf and ln. My expectation had been that drgn's behavior was more like "check in these well-known paths for a directory named after $(uname -r), then load the ELF files within". But it seems more nuanced :)