bluebuild rebase fails with error message when using podman: Missing ostree.final-diffid

[dkolb]

Running sudo bluebuild rebase ./recipes/recipe.yml results in the following error.

Rebasing onto images built via the github workflow work just fine.

[21:57:40 TRACE blue_build::commands::local:153] => rpm-ostree rebase ostree-unverified-image:oci-archive:/etc/bluebuild/bazzite-dkub.tar.gz
Pulling manifest: ostree-unverified-image:oci-archive:/etc/bluebuild/bazzite-dkub.tar.gz
error: Preparing import: Fetching manifest: Parsing manifest layout: Missing ostree.final-diffid sha256:12787d84fa137cd5649a9005efe98ec9d05ea46245fdc50aecb7dd007f2035b1
[21:57:57 ERROR blue_build::commands:36] => Failed:
  × Failed to rebase to bazzite-dkub

Repo I'm building from: https://github.com/dkolb/bazzite-dkub

Confirmed this is a podman specific issue by installing docker and running sudo bluebuild rebase -vv -B=docker ./recipes/recipe.yml.

[16:49:42 INFO  blue_build_process_management::drivers::docker_driver:338] => Successfully built image /etc/bluebuild/bazzite-dkub.tar.gz
[16:49:42 INFO  blue_build::commands::build:259] => Finished building:
    - /etc/bluebuild/bazzite-dkub.tar.gz
[16:49:42 INFO  blue_build::commands::local:151] => Rebasing image bazzite-dkub
[16:49:42 TRACE blue_build::commands::local:153] => rpm-ostree rebase ostree-unverified-image:oci-archive:/etc/bluebuild/bazzite-dkub.tar.gz
Pulling manifest: ostree-unverified-image:oci-archive:/etc/bluebuild/bazzite-dkub.tar.gz
Importing: ostree-unverified-image:oci-archive:/etc/bluebuild/bazzite-dkub.tar.gz (digest: sha256:9ba42c16d1050bab55a7f31aef27f52c8bcb91345e9eda2faee4cbfcc943e298)
ostree chunk layers already present: 72
custom layers needed: 10 (789.9 MB)
Fetching layer sha256:096521cb9543 (421 bytes)... done
Fetching layer sha256:20a81adf7440 (56.7 MB)... done
Fetching layer sha256:6f5f451934c5 (2.1 kB)... done
Fetching layer sha256:ca21fcd33873 (484.2 MB)... done
Fetching layer sha256:675f4235b360 (5.2 kB)... done
Fetching layer sha256:b7676258e2c7 (249.1 MB)... done
Fetching layer sha256:737c4c5724ba (1.0 kB)... done
Fetching layer sha256:a6fe357f9adf (4.0 kB)... done
Fetching layer sha256:f7a5f5be3971 (596 bytes)... done
Fetching layer sha256:5bd1f8fc75f7 (188 bytes)... done
Checking out tree 18847c0... done
Inactive requests:
  docker-ce (already provided by docker-ce-3:27.3.1-1.fc41.x86_64)
  docker-compose-plugin (already provided by docker-compose-plugin-2.29.7-1.fc41.x86_64)
  containerd.io (already provided by containerd.io-1.7.22-3.1.fc41.x86_64)
  docker-buildx-plugin (already provided by docker-buildx-plugin-0.17.1-1.fc41.x86_64)
  docker-ce-cli (already provided by docker-ce-cli-1:27.3.1-1.fc41.x86_64)
Staging deployment... done
Freed: 205.8 MB (pkgcache branches: 0)
Changes queued for next boot. Run "systemctl reboot" to start a reboot
[16:50:39 INFO  blue_build::commands::local:160] => Successfully rebased to bazzite-dkub

[gmpinder]

Trying things out myself. Have you tried clearing out any .tar.gz files in /etc/bluebuild/? That's where we move the bootable tarball

[gmpinder]

Ok so even though it took an insanely long time for it to build for me (one of the reasons this command is being replaced with switch #116 in v0.9.0), I was still able to rebase on my image. I'll try once more with yours and see if I get the same result as you.

[gmpinder]

Ok yeah, your build has to be doing something weird cause I get that exact same error. Could it be something with the post-build script you have?

[dkolb]

I dropped the script entirely and I'm still getting the error.

I cut back the recipe modules to just

modules:
  - type: signing

And it still gives the same error. Very strange.

[dkolb]

I've cleared out the tarball each time, also. Weird. I know you're planning a replacement for this so I'm not sure we should spend too much time on it. I'll look into seeing if switch works better.

/etc/bluebuild ro > file *.tar.gz
bazzite-dkub-buildah.tar.gz: POSIX tar archive
bazzite-dkub-docker.tar.gz:  POSIX tar archive
bazzite-dkub-podman.tar.gz:  POSIX tar archive
/etc/bluebuild ro > sha256sum *.tar.gz
02547b1c2ac734b875598f6e6c2008ec86e2da631e8166b4bcb680e844240423  bazzite-dkub-buildah.tar.gz
c0686dc00afab572fa16811451e9fa7b44bba13d2ec139671e74adacf4d80c50  bazzite-dkub-docker.tar.gz
945cdc3405bc68d0ccd799627ec7ab6a68baf1a3ef7f984aa2eaf7bb984dc062  bazzite-dkub-podman.tar.gz
/etc/bluebuild ro took 20s > ls -l *.tar.gz
-rw-r--r--. 1 root root 12204195328 Nov 11 23:20 bazzite-dkub-buildah.tar.gz
-rw-r--r--. 1 root root  4884476928 Nov 11 23:22 bazzite-dkub-docker.tar.gz
-rw-r--r--. 1 root root 12204195328 Nov 11 23:14 bazzite-dkub-podman.tar.gz
/etc/bluebuild ro > ls -lh *.tar.gz
-rw-r--r--. 1 root root  12G Nov 11 23:20 bazzite-dkub-buildah.tar.gz
-rw-r--r--. 1 root root 4.6G Nov 11 23:22 bazzite-dkub-docker.tar.gz
-rw-r--r--. 1 root root  12G Nov 11 23:14 bazzite-dkub-podman.tar.gz

Interesting that buildah/podman produce images three times the size of docker along with the build process being much, much slower. Anyway, I'm building on github and have docker installed so I can use it when I need to. Just strange that it's borked for this one build. Lol.

[dkolb]

I'm pretty sure this is a problem somewhere upstream.

~/tmp > cat Containerfile
FROM ghcr.io/ublue-os/bazzite-gnome:stable
RUN cat 'testfile' > /testfile && ostree container commit

~/tmp > sudo podman build -t localhost/bazzite-custom:local .
STEP 1/2: FROM ghcr.io/ublue-os/bazzite-gnome:stable
Trying to pull ghcr.io/ublue-os/bazzite-gnome:stable...
Getting image source signatures
Checking if image destination supports signatures
Copying blob 0724e1c33116 done  11.9MiB / 11.9MiB (skipped: 8.8KiB = 0.07%)
Copying blob 0724e1c33116 done  11.9MiB / 11.9MiB (skipped: 8.8KiB = 0.07%)
Copying blob 0724e1c33116 done  11.9MiB / 11.9MiB (skipped: 8.8KiB = 0.07%)
Copying blob 0724e1c33116 done  11.9MiB / 11.9MiB (skipped: 8.8KiB = 0.07%)
Copying blob 73ca0ed05138 done   | 
<CUT>
STEP 2/2: RUN cat 'testfile' > /testfile && ostree container commit
COMMIT localhost/bazzite-custom:local
--> 3d4e6f92835a
Successfully tagged localhost/bazzite-custom:local
3d4e6f92835a46ffd5860b66a906d5ada9746b30e789ed7f07912d6faba167dc
~/tmp took 5m13s > sudo rpm-ostree rebase ostree-unverified-image:containers-storage:localhost/bazzite-custom:local
[sudo] password for dkub: 
Pulling manifest: ostree-unverified-image:containers-storage:localhost/bazzite-custom:local
error: Preparing import: Fetching manifest: Parsing manifest layout: Missing ostree.final-diffid sha256:12787d84fa137cd5649a9005efe98ec9d05ea46245fdc50aecb7dd007f2035b1

We can probably close this issue out.

[dkolb]

Per some comments in the Universal Blue discord, buildah/podman images have to be pushed to a registry and then pulled back by rpm-ostree/bootc even if local.

podman run -d -p 5000:5000 --name local-registry registry:2

I'll test this later this week and let you know, but it is still wild your build works and mine doesn't work even for the absolute minimum build.

[gmpinder]

Per some comments in the Universal Blue discord, buildah/podman images have to be pushed to a registry and then pulled back by rpm-ostree/bootc even if local.

podman run -d -p 5000:5000 --name local-registry registry:2

I'll test this later this week and let you know, but it is still wild your build works and mine doesn't work even for the absolute minimum build.

My image is based on the normal bazzite image instead of the bazzite-gnome image. Maybe that has something to do with it?

[gmpinder]

Interesting that buildah/podman produce images three times the size of docker along with the build process being much, much slower. Anyway, I'm building on github and have docker installed so I can use it when I need to. Just strange that it's borked for this one build. Lol

Is it really that bad?? Podman/Buildah must not be handling RUN volume mounts the same way that Docker is

[dkolb]

Is it really that bad?? Podman/Buildah must not be handling RUN volume mounts the same way that Docker is

From my above comment:

/etc/bluebuild ro > ls -lh *.tar.gz
-rw-r--r--. 1 root root  12G Nov 11 23:20 bazzite-dkub-buildah.tar.gz
-rw-r--r--. 1 root root 4.6G Nov 11 23:22 bazzite-dkub-docker.tar.gz
-rw-r--r--. 1 root root  12G Nov 11 23:14 bazzite-dkub-podman.tar.gz

[gmpinder]

I did a ton of digging and testing and I was unable to get podman/buildah to create an OCI Archive of the same size as Docker. Even when using squash, it seems like it doesn't help. I don't think I can fix it from the BlueBuild CLI so I would just suggest using Docker for creating smaller archives for now. If you find a way to get the size down, I would be more than happy to implement it!

[dkolb]

Back to the original issue, it's unique to images built from bazzite at the moment. I've opened ublue-os/bazzite#1892

[gmpinder]

Back to the original issue, it's unique to images built from bazzite at the moment. I've opened ublue-os/bazzite#1892

Really? My images are based on Bazzite. I don't seem to have that issue.

[gmpinder]

Closing this as an upstream issue