segfault when "bootc install to-filesystem"

[mvo5]

When I run the attached osbuild json manifest with:

$ sudo OSBUILD_TEST_STORE=/var/tmp/osbuild-test-store PYTHONPATH=.  python3 -m osbuild  --libdir . --checkpoint image --export image --output-directory /tmp/out /tmp/bootc-install-to-fs.json

I get a segfault from bootc when using "quay.io/centos-bootc/centos-bootc-dev:stream9". Without a debug build there is almost no information so I created quay.io/rh-ee-mvogt/centos-bootc:bootc-debug (and debug2) that try to get some more data. The full strace is:

...
org.osbuild.bootc.install-to-filesystem: ac8f5579203b7f467ce49450d51bd87e75b3536958c4051a6e825612f9d12100 {}
device/disk (org.osbuild.loopback): loop1 acquired (locked: False)
mount/part4 (org.osbuild.ext4): mounting /dev/loop1p4 -> /home/egon/devel/osbuild/osbuild/.osbuild/tmp/buildroot-tmp-hnilknw3/mounts/
mount/part3 (org.osbuild.ext4): mounting /dev/loop1p3 -> /home/egon/devel/osbuild/osbuild/.osbuild/tmp/buildroot-tmp-hnilknw3/mounts/boot
mount/part2 (org.osbuild.fat): mounting /dev/loop1p2 -> /home/egon/devel/osbuild/osbuild/.osbuild/tmp/buildroot-tmp-hnilknw3/mounts/boot/efi
thread 'main' panicked at /home/mvogt/.cargo/registry/src/index.crates.io-6f17d22bba15001f/glib-0.18.5/src/gstring.rs:1957:9:
assertion failed: !ptr.is_null()
stack backtrace:
   0: rust_begin_unwind
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14
   2: core::panicking::panic
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:144:5
   3: <glib::gstring::GString as glib::translate::FromGlibPtrNone<*const u8>>::from_glib_none
             at ./home/mvogt/.cargo/registry/src/index.crates.io-6f17d22bba15001f/glib-0.18.5/src/gstring.rs:1957:9
   4: glib::translate::from_glib_none
             at ./home/mvogt/.cargo/registry/src/index.crates.io-6f17d22bba15001f/glib-0.18.5/src/translate.rs:1622:5
   5: <glib::gstring::GString as glib::translate::FromGlibPtrNone<*const i8>>::from_glib_none
             at ./home/mvogt/.cargo/registry/src/index.crates.io-6f17d22bba15001f/glib-0.18.5/src/gstring.rs:1966:9
   6: glib::translate::from_glib_none
             at ./home/mvogt/.cargo/registry/src/index.crates.io-6f17d22bba15001f/glib-0.18.5/src/translate.rs:1622:5
   7: ostree::auto::se_policy::SePolicy::name
             at ./home/mvogt/.cargo/registry/src/index.crates.io-6f17d22bba15001f/ostree-0.19.1/rust-bindings/src/auto/se_policy.rs:80:13
   8: bootc_lib::lsm::require_label::{{closure}}
             at ./home/mvogt/devel/bootc/lib/src/lsm.rs:179:17
   9: core::option::Option<T>::ok_or_else
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/option.rs:1231:25
  10: bootc_lib::lsm::require_label
             at ./home/mvogt/devel/bootc/lib/src/lsm.rs:174:5
  11: bootc_lib::lsm::selinux_ensure_install::{{closure}}
             at ./home/mvogt/devel/bootc/lib/src/lsm.rs:85:17
  12: bootc_lib::lsm::selinux_ensure_install
             at ./home/mvogt/devel/bootc/lib/src/lsm.rs:53:1
  13: bootc_lib::lsm::selinux_ensure_install_or_setenforce::{{closure}}
             at ./home/mvogt/devel/bootc/lib/src/lsm.rs:118:8
  14: bootc_lib::lsm::selinux_ensure_install_or_setenforce
             at ./home/mvogt/devel/bootc/lib/src/lsm.rs:113:1
  15: bootc_lib::install::reexecute_self_for_selinux_if_needed
             at ./home/mvogt/devel/bootc/lib/src/install.rs:804:17
  16: bootc_lib::install::prepare_install::{{closure}}
             at ./home/mvogt/devel/bootc/lib/src/install.rs:1049:9
  17: bootc_lib::install::install_to_filesystem::{{closure}}::{{closure}}
             at ./home/mvogt/devel/bootc/lib/src/install.rs:1307:87
  18: bootc_lib::install::install_to_filesystem::{{closure}}
             at ./home/mvogt/devel/bootc/lib/src/install.rs:1286:1
  19: bootc_lib::cli::run_from_opt::{{closure}}
             at ./home/mvogt/devel/bootc/lib/src/cli.rs:541:68
  20: bootc_lib::cli::run_from_iter::{{closure}}
             at ./home/mvogt/devel/bootc/lib/src/cli.rs:527:41
  21: bootc::run::{{closure}}
             at ./home/mvogt/devel/bootc/cli/src/main.rs:22:53
  22: bootc::main::{{closure}}
             at ./home/mvogt/devel/bootc/cli/src/main.rs:27:27
  23: <core::pin::Pin<P> as core::future::future::Future>::poll
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/future/future.rs:124:9
  24: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
             at ./home/mvogt/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/scheduler/current_thread/mod.rs:659:57
  25: tokio::runtime::coop::with_budget
             at ./home/mvogt/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/coop.rs:107:5
  26: tokio::runtime::coop::budget
             at ./home/mvogt/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/coop.rs:73:5
  27: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
             at ./home/mvogt/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/scheduler/current_thread/mod.rs:659:25
  28: tokio::runtime::scheduler::current_thread::Context::enter
             at ./home/mvogt/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/scheduler/current_thread/mod.rs:404:19
  29: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
             at ./home/mvogt/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/scheduler/current_thread/mod.rs:658:36
  30: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
             at ./home/mvogt/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/scheduler/current_thread/mod.rs:737:68
  31: tokio::runtime::context::scoped::Scoped<T>::set
             at ./home/mvogt/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/context/scoped.rs:40:9
  32: tokio::runtime::context::set_scheduler::{{closure}}
             at ./home/mvogt/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/context.rs:176:26
  33: std::thread::local::LocalKey<T>::try_with
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/thread/local.rs:270:16
  34: std::thread::local::LocalKey<T>::with
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/thread/local.rs:246:9
  35: tokio::runtime::context::set_scheduler
             at ./home/mvogt/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/context.rs:176:9
  36: tokio::runtime::scheduler::current_thread::CoreGuard::enter
             at ./home/mvogt/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/scheduler/current_thread/mod.rs:737:27
  37: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
             at ./home/mvogt/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/scheduler/current_thread/mod.rs:646:19
  38: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
             at ./home/mvogt/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/scheduler/current_thread/mod.rs:175:28
  39: tokio::runtime::context::runtime::enter_runtime
             at ./home/mvogt/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/context/runtime.rs:65:16
  40: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
             at ./home/mvogt/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/scheduler/current_thread/mod.rs:167:9
  41: tokio::runtime::runtime::Runtime::block_on
             at ./home/mvogt/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/runtime.rs:348:47
  42: bootc::main
             at ./home/mvogt/devel/bootc/cli/src/main.rs:27:5
  43: core::ops::function::FnOnce::call_once
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Traceback (most recent call last):
  File "/run/osbuild/bin/org.osbuild.bootc.install-to-filesystem", line 92, in <module>
    r = main(args["options"], args["inputs"], args["paths"])
  File "/run/osbuild/bin/org.osbuild.bootc.install-to-filesystem", line 87, in main
    subprocess.run(pargs, check=True, env=env)
  File "/usr/lib64/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['bootc', 'install', 'to-filesystem', '--source-imgref', 'dir:/tmp/tmpvh3p7rip/image', '--skip-fetch-check', '--generic-image', '/run/osbuild/mounts']' returned non-zero exit status 101.
mount/part2 (org.osbuild.fat): umount: /home/egon/devel/osbuild/osbuild/.osbuild/tmp/buildroot-tmp-hnilknw3/mounts/boot/efi unmounted
mount/part3 (org.osbuild.ext4): umount: /home/egon/devel/osbuild/osbuild/.osbuild/tmp/buildroot-tmp-hnilknw3/mounts/boot unmounted
mount/part4 (org.osbuild.ext4): umount: /home/egon/devel/osbuild/osbuild/.osbuild/tmp/buildroot-tmp-hnilknw3/mounts/ unmounted

⏱  Duration: 0s
manifest /tmp/bootc-install-to-fs.json failed
Failed

so it seems like some issue with getting the selinux policy label. This stacktrace is produced with a fc38 system with selinux enabled. I saw a similar one from a non-selinux system where it's also crashing on ostree::auto::se_policy::SePolicy::name

I will keep debugging this and sorry for the not very good reproduce instructions,

bootc-install-to-fs.json

[mvo5]

P.S. I wonder if centos-bootc-dev should always include a bootc debug build for better traces? but given how exceedingly rare they are maybe it's not worth the extra complexity and cost.

[cgwalters]

Yeah I saw this too, ostreedev/ostree#3218 will fix this ostree side (after a whole chain of regenerating the Rust bindings).

[mvo5]

Fwiw, it seems the fact that we get a NULL poiinter here for the SePolicy when trying to use "bootc install to-filesystem" from our osbuild buildroot means that something will not work even with the segfault itself fixed? I.e. require_labe() will then die with No label found in policy... ?

[cgwalters]

I.e. require_labe() will then die with No label found in policy... ?

Yes
#405
should have that effect.

Then we need to debug why no policy is being found.