Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

COSA container failing to build using OSBuild #5251

Closed
cgwalters opened this issue Jan 24, 2025 · 11 comments · Fixed by containers/bootc#1059 or #5253
Closed

COSA container failing to build using OSBuild #5251

cgwalters opened this issue Jan 24, 2025 · 11 comments · Fixed by containers/bootc#1059 or #5253

Comments

@cgwalters
Copy link
Member

cgwalters commented Jan 24, 2025 •
edited
Loading

https://jenkins-fedora-coreos-pipeline.apps.ocp.fedoraproject.org/blue/organizations/jenkins/build/detail/build/3319/pipeline/

The OSBuild stage is failing during the ostree container image deploy step:

[2025-01-24T21:46:31.188Z] input/images (org.osbuild.containers): target /home/jenkins/agent/workspace/build/cache/osbuild/store/tmp/buildroot-tmp-801n2htm/inputs/images

[2025-01-24T21:46:31.442Z] Failed to open file "/sys/fs/selinux/checkreqprot": Read-only file system

[2025-01-24T21:46:31.442Z] ostree container image deploy --imgref=ostree-unverified-image:oci-archive:/tmp/tmp8_jq0fh8/image --stateroot=fedora-coreos --target-imgref=ostree-image-signed:oci-archive:/fedora-coreos-42.20250124.91.0-ostree.x86_64.ociarchive --karg=rw --karg=$ignition_firstboot --karg=mitigations=auto,nosmt --sysroot=/run/osbuild/tree

[2025-01-24T21:46:36.667Z] 

[2025-01-24T21:46:36.667Z] Filesystem      Size  Used Avail Use% Mounted on

[2025-01-24T21:46:36.667Z] /dev/vdb1        30G  6.1G   24G  21% /home/jenkins/agent/workspace/build/cache

[2025-01-24T21:46:36.667Z] 

[2025-01-24T21:46:46.584Z] 

[2025-01-24T21:46:46.584Z] Filesystem      Size  Used Avail Use% Mounted on

[2025-01-24T21:46:46.584Z] /dev/vdb1        30G  6.9G   24G  23% /home/jenkins/agent/workspace/build/cache

[2025-01-24T21:46:46.584Z] 

[2025-01-24T21:46:56.503Z] /proc/self/fd/21/usr/etc/selinux/targeted/contexts/files/file_contexts.bin:  line 1 error due to: Non-ASCII characters found

[2025-01-24T21:46:56.503Z] /proc/self/fd/21/usr/etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin:  line 1 error due to: Non-ASCII characters found

[2025-01-24T21:46:56.757Z] 

[2025-01-24T21:46:56.757Z] Filesystem      Size  Used Avail Use% Mounted on

[2025-01-24T21:46:56.757Z] /dev/vdb1        30G  6.5G   24G  22% /home/jenkins/agent/workspace/build/cache

[2025-01-24T21:46:56.757Z] 

[2025-01-24T21:47:08.904Z] 

[2025-01-24T21:47:08.904Z] Filesystem      Size  Used Avail Use% Mounted on

[2025-01-24T21:47:08.904Z] /dev/vdb1        30G  6.5G   24G  22% /home/jenkins/agent/workspace/build/cache

[2025-01-24T21:47:08.904Z] 

[2025-01-24T21:47:08.904Z] /proc/self/fd/16/usr/etc/selinux/targeted/contexts/files/file_contexts.bin:  line 1 error due to: Non-ASCII characters found

[2025-01-24T21:47:08.904Z] /proc/self/fd/16/usr/etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin:  line 1 error due to: Non-ASCII characters found

[2025-01-24T21:47:08.904Z] /proc/self/fd/16/etc/selinux/targeted/contexts/files/file_contexts.bin:  line 1 error due to: Non-ASCII characters found

[2025-01-24T21:47:08.904Z] /proc/self/fd/16/etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin:  line 1 error due to: Non-ASCII characters found

[2025-01-24T21:47:08.904Z] /proc/self/fd/16/etc/selinux/targeted/contexts/files/file_contexts.bin:  line 1 error due to: Non-ASCII characters found

[2025-01-24T21:47:08.904Z] /proc/self/fd/16/etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin:  line 1 error due to: Non-ASCII characters found

[2025-01-24T21:47:09.826Z] Usage:

[2025-01-24T21:47:09.826Z]   /proc/self/exe [OPTION���] COMMAND

[2025-01-24T21:47:09.826Z] 

[2025-01-24T21:47:09.826Z] Builtin Commands:

[2025-01-24T21:47:09.826Z]   apply-live             Apply pending deployment changes to booted deployment

[2025-01-24T21:47:09.826Z]   cancel                 Cancel an active transaction

[2025-01-24T21:47:09.826Z]   cleanup                Clear cached/pending data

[2025-01-24T21:47:09.826Z]   compose                Commands to compose a tree

[2025-01-24T21:47:09.826Z]   db                     Commands to query the RPM database

[2025-01-24T21:47:09.826Z]   deploy                 Deploy a specific commit

[2025-01-24T21:47:09.826Z]   initramfs              Enable or disable local initramfs regeneration

[2025-01-24T21:47:09.826Z]   initramfs-etc          Add files to the initramfs

[2025-01-24T21:47:09.826Z]   install                Overlay additional packages

[2025-01-24T21:47:09.826Z]   kargs                  Query or modify kernel arguments

[2025-01-24T21:47:09.826Z]   override               Manage base package overrides

[2025-01-24T21:47:09.826Z]   rebase                 Switch to a different tree

[2025-01-24T21:47:09.826Z]   refresh-md             Generate rpm repo metadata

[2025-01-24T21:47:09.826Z]   reload                 Reload configuration

[2025-01-24T21:47:09.826Z]   reset                  Remove all mutations

[2025-01-24T21:47:09.826Z]   rollback               Revert to the previously booted tree

[2025-01-24T21:47:09.826Z]   search                 Search for packages

[2025-01-24T21:47:09.826Z]   status                 Get the version of the booted system

[2025-01-24T21:47:09.826Z]   uninstall              Remove overlayed additional packages

[2025-01-24T21:47:09.826Z]   upgrade                Perform a system upgrade

[2025-01-24T21:47:09.826Z]   usroverlay             Apply a transient overlayfs to /usr

[2025-01-24T21:47:09.826Z] 

[2025-01-24T21:47:09.826Z] Help Options:

[2025-01-24T21:47:09.826Z]   -h, --help       Show help options

[2025-01-24T21:47:09.826Z] 

[2025-01-24T21:47:09.826Z] Application Options:

[2025-01-24T21:47:09.826Z]   --version        Print version information and exit

[2025-01-24T21:47:09.826Z]   -q, --quiet      Avoid printing most informational messages

[2025-01-24T21:47:09.826Z] 

[2025-01-24T21:47:09.826Z] error: Unknown command 'internals'

[2025-01-24T21:47:09.826Z] error: Performing deployment: Failed to complete bootc install

[2025-01-24T21:47:09.826Z] Traceback (most recent call last):

[2025-01-24T21:47:09.826Z]   File "/run/osbuild/bin/org.osbuild.ostree.deploy.container", line 72, in <module>

[2025-01-24T21:47:09.826Z]     r = main(stage_args["tree"],

[2025-01-24T21:47:09.826Z]              stage_args["inputs"],

[2025-01-24T21:47:09.826Z]              stage_args["options"])

[2025-01-24T21:47:09.826Z]   File "/run/osbuild/bin/org.osbuild.ostree.deploy.container", line 67, in main

[2025-01-24T21:47:09.826Z]     ostree_container_deploy(tree, inputs, osname, target_imgref, kopts)

[2025-01-24T21:47:09.826Z]     ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

[2025-01-24T21:47:09.826Z]   File "/run/osbuild/bin/org.osbuild.ostree.deploy.container", line 41, in ostree_container_deploy

[2025-01-24T21:47:09.826Z]     ostree.cli("container", "image", "deploy",

[2025-01-24T21:47:09.826Z]     ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

[2025-01-24T21:47:09.826Z]                *extra_args, sysroot=tree, *kargs)

[2025-01-24T21:47:09.826Z]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

[2025-01-24T21:47:09.826Z]   File "/run/osbuild/lib/osbuild/util/ostree.py", line 226, in cli

[2025-01-24T21:47:09.826Z]     return subprocess.run(["ostree"] + args,

[2025-01-24T21:47:09.826Z]            ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^

[2025-01-24T21:47:09.826Z]                           encoding="utf8",

[2025-01-24T21:47:09.826Z]                           ^^^^^^^^^^^^^^^^

[2025-01-24T21:47:09.826Z]                           stdout=subprocess.PIPE,

[2025-01-24T21:47:09.826Z]                           ^^^^^^^^^^^^^^^^^^^^^^^

[2025-01-24T21:47:09.826Z]                           input=_input,

[2025-01-24T21:47:09.826Z]                           ^^^^^^^^^^^^^

[2025-01-24T21:47:09.826Z]                           check=True)

[2025-01-24T21:47:09.826Z]                           ^^^^^^^^^^^

[2025-01-24T21:47:09.826Z]   File "/usr/lib64/python3.13/subprocess.py", line 577, in run

[2025-01-24T21:47:09.826Z]     raise CalledProcessError(retcode, process.args,

[2025-01-24T21:47:09.826Z]                              output=stdout, stderr=stderr)

[2025-01-24T21:47:09.826Z] subprocess.CalledProcessError: Command '['ostree', 'container', 'image', 'deploy', '--imgref=ostree-unverified-image:oci-archive:/tmp/tmp8_jq0fh8/image', '--stateroot=fedora-coreos', '--target-imgref=ostree-image-signed:oci-archive:/fedora-coreos-42.20250124.91.0-ostree.x86_64.ociarchive', '--karg=rw', '--karg=$ignition_firstboot', '--karg=mitigations=auto,nosmt', '--sysroot=/run/osbuild/tree']' returned non-zero exit status 1.

As of recently the ostree container command redirects to bootc. Could we somehow not have bootc or a tool old bootc in this image?
@cgwalters
Copy link
Member Author

This relates to containers/bootc#915 likely

@dustymabe
Copy link
Member

dustymabe commented Jan 24, 2025 •
edited
Loading 

[dustymabe@media ~]$ podman run -it --rm --entrypoint /bin/bash quay.io/coreos-assembler/coreos-assembler:bad-20240124
[builder@f5be8adfe807 srv]$ rpm -q bootc rpm-ostree
bootc-1.1.4-6.fc41.x86_64
rpm-ostree-2025.2-1.fc41.x86_64

@dustymabe dustymabe changed the title recent rawhide build COSA container failing to build using OSBuild Jan 24, 2025
@cgwalters
Copy link
Member Author

And um...this is running osbuild inside supermin right?

@dustymabe
Copy link
Member

And um...this is running osbuild inside supermin right?

Yes

@cgwalters
Copy link
Member Author

OK right we merged #5227 which was written assuming that the invoking process was bootc but here it's rpm-ostree

@cgwalters cgwalters mentioned this issue Jan 24, 2025
Merged
@cgwalters cgwalters transferred this issue from coreos/fedora-coreos-tracker Jan 24, 2025
cgwalters added a commit to cgwalters/bootc that referenced this issue Jan 24, 2025
@cgwalters
ostree-ext: Add bootc feature
57111be 
This is aiming to fix coreos/rpm-ostree#5251
Basically we can't re-invoke the calling process if we aren't
actually bootc.

(There are solutions that make this work, but what we'll hopefully
 do shortly is propagate the logic to older streams to have
 bootc take over the handling anyways)

Signed-off-by: Colin Walters <walters@verbum.org>
@cgwalters cgwalters mentioned this issue Jan 24, 2025
Merged
@cgwalters
Copy link
Member Author

OK us vendoring a bootc with containers/bootc#1059 should fix this, testing

@cgwalters
Copy link
Member Author

The other fix would be to push through https://github.com/containers/bootc/blob/6ff180b56a2ff430ee4ac94c9a089a936fec3f4e/contrib/packaging/bootc.spec#L2 to f41, but eh

@cgwalters
Copy link
Member Author

The FCOS CI on rpm-ostree should have caught this as far as I know...I don't understand how it didn't. We're deploying the new rpm-ostree on top of existing coreos-assembler which should be this same scenario.

@dustymabe
Copy link
Member

The FCOS CI on rpm-ostree should have caught this as far as I know...I don't understand how it didn't. We're deploying the new rpm-ostree on top of existing coreos-assembler which should be this same scenario.

Yeah. I was wondering about this too.

cgwalters added a commit to cgwalters/rpm-ostree that referenced this issue Jan 24, 2025
@cgwalters
cci: Install built rpm on top of cosa
4276fea 
I would have sworn we were doing this before, but if so I can't
find it in git logs.

We want to test the new rpm-ostree at build time too of course.

xref coreos#5251 (comment)
@cgwalters cgwalters mentioned this issue Jan 24, 2025
Closed
@cgwalters
Copy link
Member Author

okay now I'm getting

evice/disk (org.osbuild.loopback): loop0 acquired (locked: False)
mount/root (org.osbuild.xfs): mounting /dev/loop0p4 -> /var/srv/walters/builds/fcos/cache/osbuild/store/tmp/buildroot-tmp-beaxmuyk/mounts/
mount/boot (org.osbuild.ext4): mounting /dev/loop0p3 -> /var/srv/walters/builds/fcos/cache/osbuild/store/tmp/buildroot-tmp-beaxmuyk/mounts/boot
mount/efi (org.osbuild.fat): mounting /dev/loop0p2 -> /var/srv/walters/builds/fcos/cache/osbuild/store/tmp/buildroot-tmp-beaxmuyk/mounts/boot/efi
Failed to open file "/sys/fs/selinux/checkreqprot": Read-only file system
copying '/run/osbuild/inputs/tree/.' -> '/run/osbuild/mounts/.'
mount/efi (org.osbuild.fat): umount: /var/srv/walters/builds/fcos/cache/osbuild/store/tmp/buildroot-tmp-beaxmuyk/mounts/boot/efi unmounted
mount/boot (org.osbuild.ext4): sync: error syncing '/var/srv/walters/builds/fcos/cache/osbuild/store/tmp/buildroot-tmp-beaxmuyk/mounts/boot': Input/output error

Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb1        20G   20G   28K 100% /var/srv/walters/builds/fcos/cache

Okay guess I need to nuke cache.qcow2

@cgwalters
Copy link
Member Author

OK right we merged #5227 which was written assuming that the invoking process was bootc but here it's rpm-ostree

And to be clear this should all work in rawhide/c10s where bootc owns ostree container.

Anyways I can confirm containers/bootc#1059 fixes this.

@cgwalters cgwalters reopened this Jan 25, 2025
@cgwalters cgwalters mentioned this issue Jan 25, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

ostree-ext: Add bootc feature cgwalters/bootc
Update bootc cgwalters/rpm-ostree
2 participants
@cgwalters @dustymabe