Arch Linux = error: failed retrieving file 'core.db' from geo.mirror.pkgbuild.com : Could not resolve host: geo.mirror.pkgbuild.com

[componentscience]

mkosi commit the issue has been seen with

main

Used host distribution

Arch Linux 2024.07.01

Used target distribution

Arch Linux 2024.07.01

Linux kernel version used

6.9.9-arch1-1

CPU architectures issue was seen on

x86_64

Unexpected behaviour you saw

When running a vanilla # mkosi command, the error below is thrown. There are no DNS issues on this system that we know of.

[root@myhost mydir]# mkosi
Create subvolume '/var/tmp/mkosi-workspace-0bya2h_o/root'
‣ Syncing package manager metadata for default image
:: Synchronizing package databases...
core.db failed to download
extra.db failed to download
error: failed retrieving file 'core.db' from geo.mirror.pkgbuild.com : Could not resolve host: geo.mirror.pkgbuild.com
warning: fatal error from geo.mirror.pkgbuild.com, skipping for the remainder of this transaction
error: failed retrieving file 'extra.db' from geo.mirror.pkgbuild.com : Could not resolve host: geo.mirror.pkgbuild.com
error: failed to synchronize all databases (invalid url for server)
‣ "pacman --root=/buildroot --logfile=/dev/null --dbpath=/var/lib/pacman --cachedir=/var/cache/pacman/mkosi --cachedir=/var/cache/pacman/pkg --hookdir=/buildroot/etc/pacman.d/hooks --arch x86_64 --color auto --noconfirm --sync --refresh" returned non-zero exit code 1.

Used mkosi config

The vanilla # mkosi command was ran without any config files as a test.

mkosi output

[root@myhost mydir]# mkosi --debug
‣ + stat --file-system --format %T /var/tmp/mkosi-workspace-m7patsrx
‣ + btrfs subvolume create /var/tmp/mkosi-workspace-m7patsrx/root
Create subvolume '/var/tmp/mkosi-workspace-m7patsrx/root'
‣ Syncing package manager metadata for default image
‣ Acquiring lock on /var/cache/mkosi/arch~rolling~x86-64/cache/pacman
‣ Acquired lock on /var/cache/mkosi/arch~rolling~x86-64/cache/pacman
‣ Acquiring lock on /var/cache/mkosi/arch~rolling~x86-64/lib/pacman
‣ Acquired lock on /var/cache/mkosi/arch~rolling~x86-64/lib/pacman
‣ + pacman --root=/buildroot --logfile=/dev/null --dbpath=/var/lib/pacman --cachedir=/var/cache/pacman/mkosi --cachedir=/var/cache/pacman/pkg --hookdir=/buildroot/etc/pacman.d/hooks --arch x86_64 --color auto --noconfirm --sync --refresh
:: Synchronizing package databases...
 core.db failed to download
 extra.db failed to download
error: failed retrieving file 'core.db' from geo.mirror.pkgbuild.com : Could not resolve host: geo.mirror.pkgbuild.com
warning: fatal error from geo.mirror.pkgbuild.com, skipping for the remainder of this transaction
error: failed retrieving file 'extra.db' from geo.mirror.pkgbuild.com : Could not resolve host: geo.mirror.pkgbuild.com
error: failed to synchronize all databases (invalid url for server)
‣ "bwrap --die-with-parent --proc /proc --setenv SYSTEMD_OFFLINE 1 --unsetenv TMPDIR --tmpfs /tmp --unshare-ipc --dev /dev --symlink usr/bin /bin --symlink usr/bin /sbin --symlink usr/lib /lib --symlink usr/lib /lib64 --setenv PATH /scripts:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl --uid 0 --gid 0 --cap-add ALL --dir /work/src --chdir /work/src --bind /var/tmp/mkosi-workspace-m7patsrx/root /buildroot --bind /var/tmp/mkosi-workspace-m7patsrx/pkgmngr/etc /etc --ro-bind /etc/ca-certificates /etc/ca-certificates --ro-bind /etc/pacman.d/gnupg /etc/pacman.d/gnupg --bind /etc/resolv.conf /etc/resolv.conf --ro-bind /etc/ssl /etc/ssl --ro-bind /etc/ca-certificates/extracted/tls-ca-bundle.pem /proxy.cacert --ro-bind /usr /usr --bind '/var/cache/mkosi/arch~rolling~x86-64/cache/pacman' /var/cache/pacman --ro-bind /var/tmp/mkosi-workspace-m7patsrx/packages /var/cache/pacman/mkosi --bind '/var/cache/mkosi/arch~rolling~x86-64/lib/pacman' /var/lib/pacman --bind /var/tmp/mkosi-workspace-m7patsrx/root/var/lib/pacman/local /var/lib/pacman/local --bind /var/tmp/mkosi-workspace-m7patsrx/pkgmngr/var/log /var/log --bind /var/tmp/mkosi-workspace-m7patsrx/tmp/mkosi-var-tmp-eb0dab030aaa4b9f /var/tmp --bind /var/tmp/mkosi-workspace-m7patsrx/packages /work/packages --bind /nspawn /work/src sh -c 'chmod 1777 /dev/shm && chmod 755 /etc && exec $0 "$@"' setpgid --foreground -- pacman --root=/buildroot --logfile=/dev/null --dbpath=/var/lib/pacman --cachedir=/var/cache/pacman/mkosi --cachedir=/var/cache/pacman/pkg --hookdir=/buildroot/etc/pacman.d/hooks --arch x86_64 --color auto --noconfirm --sync --refresh" returned non-zero exit code 1.
‣ + rm -rf -- /var/tmp/mkosi-workspace-m7patsrx
Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/mkosi/run.py", line 60, in uncaught_exception_handler
    yield
  File "/usr/lib/python3.12/site-packages/mkosi/run.py", line 101, in fork_and_wait
    target(*args, **kwargs)
  File "/usr/lib/python3.12/site-packages/mkosi/__init__.py", line 4566, in run_sync
    sync_repository_metadata(context)
  File "/usr/lib/python3.12/site-packages/mkosi/__init__.py", line 4534, in sync_repository_metadata
    context.config.distribution.package_manager(context.config).sync(context)
  File "/usr/lib/python3.12/site-packages/mkosi/installer/pacman.py", line 183, in sync
    cls.invoke(
  File "/usr/lib/python3.12/site-packages/mkosi/installer/pacman.py", line 166, in invoke
    return run(
           ^^^^
  File "/usr/lib/python3.12/site-packages/mkosi/run.py", line 150, in run
    with spawn(
  File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/usr/lib/python3.12/site-packages/mkosi/run.py", line 351, in spawn
    raise subprocess.CalledProcessError(returncode, cmdline)
subprocess.CalledProcessError: Command '['pacman', '--root=/buildroot', '--logfile=/dev/null', '--dbpath=/var/lib/pacman', '--cachedir=/var/cache/pacman/mkosi', '--cachedir=/var/cache/pacman/pkg', '--hookdir=/buildroot/etc/pacman.d/hooks', '--arch', 'x86_64', '--color', 'auto', '--noconfirm', '--sync', '--refresh']' returned non-zero exit status 1.
‣ + tput cnorm
‣ + tput smam

[DaanDeMeyer]

@componentscience Can you run with --debug-shell and check if /etc/resolv.conf looks correct within the sandbox for your system?

[componentscience]

@DaanDeMeyer after specifying a nameserver in /etc/resolv.conf everything is working. Previously we had only configured /etc/systemd/resolved.conf. Closing this. Thank you.

[edorgeville]

For anyone else with a similar DNS resolution issue in their scripts and your host is WSL2, the issue might be caused by this:
https://learn.microsoft.com/en-us/windows/wsl/wsl-config#network-settings
By default, WSL2 symlinks its /etc/resolv.conf to an automatically generated file. Disabling this and writing the WSL2's  /etc/resolv.conf manually might solve the issue, as explained by this comment: microsoft/WSL#5420 (comment)
In my case, as I am using systemd-resolved in WSL2, I had to edit /etc/systemd/resolved.conf and add a DNS server here. Then, I restarted resolved by doing sudo systemctl restart systemd-resolved.

Another workaround is to override /etc/resolv.conf in chrooted scripts:

#!/bin/sh
# ./mkosi.build.chroot

rm /etc/resolv.conf
echo "nameserver 8.8.8.8" > /etc/resolv.conf

# DNS resolution fixed!

EDIT:
I just discovered that an image without systemd-resolved under its Packages doesn't cause the issue. The following log during the build shows the symlink taking over the chroot sandbox bind:

Converting /etc/resolv.conf to a symlink to /run/systemd/resolve/stub-resolv.conf...

Updated workaround that works both with and without systemd-resolved installed:

#!/bin/sh
# ./mkosi.build.chroot

# If systemd-resolved was installed before build,
# resolv.conf is a dead symlink, so we write our own.
# This does not get persisted after the build step.
if [ -L /etc/resolv.conf ]; then
  rm /etc/resolv.conf
  echo "nameserver 8.8.8.8" > /etc/resolv.conf
fi

# DNS resolution fixed!