Can't get network connection on Hetzner dedicated server with two NICs

[deitmerit]

I am trying to install Talos on three Hetzner dedicated servers, all of which have two NICs. One is connected to the public internet, the other one connects to a dedicated switch that provides a private network between the three servers.

These are the steps I tried:

• boot server in rescue mode and SSH to it
• download the metal-amd64.raw.xz image, write it do disk, and reboot:
  wget -O- https://github.com/siderolabs/talos/releases/download/v1.7.6/metal-amd64.raw.xz | unxz --stdout - | dd of=/dev/nvme0n1 && sync && reboot
• try talosctl -n <server IP address> disks --insecure and get:
  error getting disks: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [ip redacted]:50000: i/o timeout"

When I connect the vKVM rescue system, I can see that Talos starts up, but cannot connect to the Internet, as there is no network configured ("CONNECTIVITY FAILED", no IP address). When trying to manually configure the network (F3), I can only select one NIC (ens4), which seems to be the "internal" NIC. I tried setting a static address there, both the public IP of the server and a private IP didn't change anything - I still can't talosctl or even ping the machine, neither via the public IP nor via the private IP from one of the other servers.

This is what the rescue system tells me about the NICs:

eth0  LINK: no
MAC:  [redacted]
IP:   (none)
IPv6: (none)
Intel(R) PRO/1000 Network Driver
eth1  LINK: yes
MAC:  [redacted]
IP:   [redacted]
IPv6: [redacted]
RealTek RTL-8169 Gigabit Ethernet driver

This is the output of lspci:

00:00.0 Host bridge: Intel Corporation Device 4640 (rev 02)
00:01.0 PCI bridge: Intel Corporation 12th Gen Core Processor PCI Express x16 Controller #1 (rev 02)
00:02.0 VGA compatible controller: Intel Corporation AlderLake-S GT1 (rev 0c)
00:06.0 PCI bridge: Intel Corporation 12th Gen Core Processor PCI Express x4 Controller #0 (rev 02)
00:0a.0 Signal processing controller: Intel Corporation Platform Monitoring Technology (rev 01)
00:14.0 USB controller: Intel Corporation Raptor Lake USB 3.2 Gen 2x2 (20 Gb/s) XHCI Host Controller (rev 11)
00:14.2 RAM memory: Intel Corporation Raptor Lake-S PCH Shared SRAM (rev 11)
00:15.0 Serial bus controller: Intel Corporation Raptor Lake Serial IO I2C Host Controller #0 (rev 11)
00:16.0 Communication controller: Intel Corporation Raptor Lake CSME HECI #1 (rev 11)
00:17.0 SATA controller: Intel Corporation Raptor Lake SATA AHCI Controller (rev 11)
00:1a.0 PCI bridge: Intel Corporation Raptor Lake PCI Express Root Port #25 (rev 11)
00:1c.0 PCI bridge: Intel Corporation Raptor Lake PCI Express Root Port #1 (rev 11)
00:1c.2 PCI bridge: Intel Corporation Device 7a3a (rev 11)
00:1d.0 PCI bridge: Intel Corporation Device 7a36 (rev 11)
00:1d.7 PCI bridge: Intel Corporation Device 7a37 (rev 11)
00:1f.0 ISA bridge: Intel Corporation Device 7a06 (rev 11)
00:1f.4 SMBus: Intel Corporation Raptor Lake-S PCH SMBus Controller (rev 11)
00:1f.5 Serial bus controller: Intel Corporation Raptor Lake SPI (flash) Controller (rev 11)
01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO
02:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO
05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
07:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection

I have tried generating custom images via factory.talos.dev, and set kernel parameters like net.ifnames=0 (which changes the name of the available NIC to eth0), talos.network.interface.ignore to skip the internal NIC, ip=... to explicitly configure both interfaces, but all to no avail (I can get CONNECTIVITY OK, but still can't connect from anywhere).

When scrolling to the log output of Talos / Linux, I only see messages about "Intel(R) PRO/1000", but not about "RealTek RTL-8169", so I'm wondering if the OS even detects the RealTek NIC.

The kernel parameters of the Hetzner rescue system are:

root@rescue ~ # cat /proc/cmdline
vmlinuz initrd=initrd.img nfsdir=[redacted]:/nfs RFILE=rescue-amd64-bookworm-v008.ext2 HASH=[redacted] net.ifnames=0 vga=0x317 lang=de quiet IP6=[redacted] IP6MASK=64 IP6GW=fe80::1 config_url=https://admin-robot.your-server.de/admin/api/boot/rescue/[redacted] keyboard=de BOOTIF=01-[MAC addr redacted]

When I manually configure the Intel NIC in the rescue system to use some private IP, I can ping it from the other two servers, so both NICs work as expected there.

So, my impression is that Talos can't find the RealTek NIC. How can I fix this, or at least find out what the problem is?

[smira]

I have no specific answer to your question, but you could try an image with realtek-firmware included

[TimJones]

Have you contacted Hetzner support just to confirm that their rescue system runs on the same physical hardware that it is rescuing? Share with them the dmesg/lspci discrepancies and see if they know why it doesn't show the same info?

[deitmerit]

I have put a lot of time into this today and have found out a few things.
We have to distinguish between the Hetzner rescue system and the Hetzner vKVM system.
The rescue system is a PXE booted Ubuntu that runs on the physical hardware.
The vKVM system also boots via PXE, starts a vm via KVM, and runs the system that is installed on the physical disk in that vm (see https://docs.hetzner.com/robot/dedicated-server/virtualization/vkvm/).

One thing that I didn't go into detail about because I thought it would only confuse the discussion, but now turned out to be essential:
When I installed the raw Talos image to disk and rebooted, I never got a network connection, with neither of the NICs. When I started the vKVM, I could connect via the Intel NIC. I didn't realize that this meant that Talos would run in a vm. So, the output of talosctl get pcidevices is in fact not the physical hardware, but the view from inside the vm!

Hetzner also offers to connect a physical kvm console, which allows me to see the actual screen without having to run a vm. And with this kvm console, I today realized that Talos doesn't boot at all when run on the physical hardware. It boots into the grub menu, and then only one line is written to the screen:

EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA__GUID device path

When I turn on "Security Device Support" in the BIOS, another line turns up:

EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA__GUID device path
EFI stub: Measured initrd data into PCR 9

Then the machine hangs without further output.

So, I know that this is a completely different topic than I originally thought, and I'm very sorry for the confusion. The fact that the Realtek NIC doesn't work still holds, but only in the Hetzner vKVM setup. It's possible that it works when running directly on the physical hardware.

The new question is: do you have any idea what could cause the boot process to get stuck after loading the initrd? What can I do to find out more about the cause of this? I tried different images (metal, nocloud, with and without secure boot), different versions (from 1.4.4 to 1.8.0-alpha.1, with the pre-1.7 versions not even showing the single initrd line), tried getting more output via the earlycon kernel parameter, but all to no avail.

[deitmerit]

Addendum: I have had Hetzner update the BIOS on all three systems to the newest version, but that didn't fix the problem.

[deitmerit]

We found a fix! Removing Kernel parameter console=ttyS0 fixes both problems - Talos boots on the physical hardware, and the Realtek onboard NIC works!