rpm-ostree-toolbox creates ostree tree with missing grub file, resulting in a failing installer

[leandrosansilva]

Hello, I am using rpm-ostree-toolbox to create a customized version of CentOS 7. I am using Fedora 22 as the building host. I try to create an installed, but the installation fails due a missing file related to grub. And this file is supposed to be in the tree, but is not indeed. The installer complains about a file /usr/lib/ostree-boot/grub2/grubenv which is a link to /boot/efi/EFI/centos/grubenv, but this one does not exist and I can check it using "ostree ls".

I have generated the repository using rpm-ostree with the following command (the files are attached to this ticket):

# rpm-ostree-toolbox treecompose -c centos-atomic-egym.ini --ostreerepo repo

It creates a directory repo and in my case I check first the link:
# ostree --repo=repo ls centos-atomic-egym/7/x86_64/standard /usr/lib/ostree-boot/grub2
d00755 0 0 0 /usr/lib/ostree-boot/grub2
l00777 0 0 0 /usr/lib/ostree-boot/grub2/grubenv -> /boot/efi/EFI/centos/grubenv
d00755 0 0 0 /usr/lib/ostree-boot/grub2/themes

Then I check for the file /boot/efi/EFI/centos/grubenv:
# ostree --repo=repo ls centos-atomic-egym/7/x86_64/standard /boot/efi/EFI/centos/grubenv
error: No such file or directory: /boot/efi/EFI/centos/grubenv

The directory where it is contained:
# ostree --repo=repo ls centos-atomic-egym/7/x86_64/standard /boot/efi/EFI/centos
d00755 0 0 0 /boot/efi/EFI/centos
-00644 0 0 128 /boot/efi/EFI/centos/BOOT.CSV
-00644 0 0 1268168 /boot/efi/EFI/centos/MokManager.efi
-00755 0 0 1070976 /boot/efi/EFI/centos/gcdx64.efi
-00755 0 0 1070976 /boot/efi/EFI/centos/grubx64.efi
-00644 0 0 1376112 /boot/efi/EFI/centos/shim-redhat.efi
-00644 0 0 1384792 /boot/efi/EFI/centos/shim.efi
d00755 0 0 0 /boot/efi/EFI/centos/fonts

This ref revision is 8d64657fcf24070f9ceace0b4332eea114ab1df2dc1bded3c03647c6a27e22ee.
The error when the installer fails is:

Error: [('/mnt/sysimage/ostree/deploy/centos-atomic-egym/deploy/8d64657fcf24070f9ceace0b4332eea114ab1df2dc1bded3c03647c6a27e22ee.0/usr/lib/ostree-boot/grub2/grubenv', '/mnt/sysimage/boot/grub2/grubenv', "[Errno 2] No such file or directory: '/mnt/sysimage/ostree/deploy/centos-atomic-egym/deploy/8d64657fcf24070f9ceace0b4332eea114ab1df2dc1bded3c03647c6a27e22ee.0/usr/lib/ostree-boot/grub2/grubenv'")].

The configuration files and error messages generated by the installer can be found at:
https://github.com/leandrosansilva/custom-centos-atomic-builder

Thank you in advance.

ps: please let me know if I should have reported this bug to the centos developers.

[mainframe]

Im reproducing the same issue - when trying to use install.iso - built from unmodified https://github.com/CentOS/sig-atomic-buildscripts configuration set.

Error screenshot from iso install VM console: http://prntscr.com/7lzvro

Also Im not able to build successfully kvm image - anaconda install process in imagefactory VM fails with the following error: http://prntscr.com/7lq38e

From the build VM /tmp/anaconda.log:

21:40:35,454 INFO anaconda: Creating xfs on /dev/vda1
21:40:36,242 INFO anaconda: executing ostreesetup=<pykickstart.commands.ostreesetup.RHEL7_OSTreeSetup object at 0x7fdd038d3110>
21:40:36,570 ERR anaconda: Failed to pull from repository: Server returned status 404: Not Found

Last lines from the build VM /tmp/program.log:

21:40:36,339 INFO program: Running... ostree --repo=/mnt/sysimage/ostree/repo remote add --set=gpg-verify=false installmedia http://192.168.122.1:47990/
21:40:36,377 DEBUG program: Return code: 0

[cgwalters]

@mainframe That looks like you don't have an ostree commit in the repo, or the repo wasn't found. See https://github.com/CentOS/sig-atomic-buildscripts/blob/downstream/build_ostree_components.sh#L63

[cgwalters]

@leandrosansilva This is fixed by rhinstaller/anaconda@d4cd779 I'm pretty sure. What version of Anaconda are you using?

[mainframe]

@cgwalters I was overlooking downstream branch. Still Im stuck we some new problems:

• for the sake of correctness - HomeDir variable is missing from build_ostree_components.sh script - so working example inside that script could be something like this:
  HomeDir=/srv ./build_ostree_components.sh /srv/builddir
• HomeDir=/srv ./build_ostree_components.sh /srv/builddir currently fails with the following error:
  Installing packages: 14%
  error: Error running transaction: file /usr/bin/docker-storage-setup conflicts between attemped installs of docker-storage-setup-0.0.4-2.el7.noarch

[leandrosansilva]

@cgwalters not sure, It's the default installer of Fedora22 cloud, but I also tested from fedora raw from the same day I posted this report.

[mainframe]

@cgwalters - recent updates to sig-atomic-buildscripts fixed tree building docker-storage-setup related error - however Im back at square one - as ostreesetup by kickstart inside imagefactory VM still fails to access ostreesetup repo over 192.168.122.1 virbr0 - this time saying "connection refused". However if I then switch to anaconda shell console inside this VM - I can do curl http://192.168.122.1/repo/ and to get succesful directory listing. There is no indication what was the exact URL tried by anaconda - just kickstart contains --url="http://192.168.122.1:8000/repo/" - so Im guessing that this fails. Any ideas what might be wrong still?

[mainframe]

Solved the "connection refused" issue - seems that it was complaining about internet connection from imagefactory build VM - which was blocked because shutting down firewalld changed libvirt iptables ruleset (and libvirtd wasnt restarted after shutting down firewalld). Im still puzzled why image build process requires outside connection - if ostree repo should be coming from host (192.168.122.1)?

[cgwalters]

rpm-ostree-toolbox has been in a tension for a while between "useful for developers on one machine" and "rel-eng tool". In a more production rel-eng scenario, the idea is that the tree compose happens on a separate machine, and one would point to it in the kickstart file with real DNS etc.