Macs with BCM57765 Ethernet Controller: System unresponsive with network cable plugged in

[probonopd]

Discussed in helloSystem/hello#297

^{Originally posted by patmaddox December 23, 2021}
I'm trying to get started with 0.7 on a 2013 iMac. Something seems to not like my NIC (which uses bge driver according to ifconfig).

Can't interact with UI when booting with network cable plugged in (doesn't finish booting?)

If I boot from USB with the network cable plugged in, it seems to not finish booting. I see the desktop, and an icon called Starvolume, but I don't see Live or EFI (which is what I see when I boot without a cable. The mouse and keyboard are non-responsive.

Attaching network cable after booting makes UI non-interactive

If I boot from USB without the cable plugged in, I get a responsive UI and can open applications. I've opened the Terminal and looked at ifconfig. After I plug the network cable in, I can still use the Terminal and see that the NIC has been configured via DHCP.

Next I try to launch an application - Create Live Media so I can install helloSystem onto the internal hard drive. At this point I'm unable to interact with the UI at all. I can move the mouse cursor around, but not click anything, and the keyboard is unresponsive. I can no longer interact with Terminal either.

---

I don't have a clue how to debug this since I can't interact with anything after plugging in the network cable. Any ideas?

[probonopd]

On Mac mini Mid 2010 (Macmini4,1) (which contains the NetXtreme BCM57765 Gigabit Ethernet PCIe) helloSystem 0.7.0 becomes unresponsive then the Ethernet cable is plugged in. If the Logs utility is open, the last message seen in /var/log/messages is bge0: watchdog timeout --resetting.

Even with dev.bge.0.msi=0 in /boot/loader.conf, the system becomes unresponsive as soon as the Ethernet cable is plugged in.

Logs:

References

• https://forums.freebsd.org/threads/bge0-watchdog-timeout-resetting.39619/

[probonopd]

According to https://www.truenas.com/community/threads/error-bge0-watchdog-timeout-resetting.64345/, hw.bge.allow_asf=0 is needed. When entering this manually at the bootloader prompt (in addition to dev.bge.0.msi=0), then booting the helloSystem Live ISO, then plugging in the network cable, the system reboots after a few seconds! If booting with the network cable already plugged in, the desktop is unresponsive from the beginning.

[probonopd]

Possibly related: [zone: mbuf_cluster] kern.ipc.nmbufs limit reached appears immediately after "bge0: link state changed to UP`.

https://forum.netgate.com/topic/95822/kern-ipc-nmbufs-limit-reached/9 points to https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards which suggests additional loader.conf entries.

Using:

dev.bge.0.msi=0
hw.bge.allow_asf=0
kern.ipc.nmbclusters="1000000"
hw.bce.tso_enable=0
hw.pci.enable_msix=0

Results in the system not crashing but network to being not functional.

The interface gets an IP address using DHCP but DNS resolution does not work and pinging it from another device leads to massive packet loss:

FreeBSD% ping 192.168.0.178
PING 192.168.0.178 (192.168.0.178): 56 data bytes
64 bytes from 192.168.0.178: icmp_seq=0 ttl=64 time=0.638 ms
64 bytes from 192.168.0.178: icmp_seq=3 ttl=64 time=0.697 ms
64 bytes from 192.168.0.178: icmp_seq=6 ttl=64 time=0.661 ms
64 bytes from 192.168.0.178: icmp_seq=9 ttl=64 time=1.132 ms
64 bytes from 192.168.0.178: icmp_seq=10 ttl=64 time=0.744 ms
64 bytes from 192.168.0.178: icmp_seq=13 ttl=64 time=0.746 ms
^C
--- 192.168.0.178 ping statistics ---
16 packets transmitted, 6 packets received, 62.5% packet loss

dmesg is showing

bge0: link state changed to UP
bge0: link state changed to DOWN
bge0: link state changed to UP
bge0: link state changed to DOWN
bge0: link state changed to UP
bge0: link state changed to DOWN
bge0: link state changed to UP
bge0: link state changed to DOWN
...

Same result when only using

dev.bge.0.msi=0
kern.ipc.nmbclusters="1000000"

[probonopd]

https://lists.freebsd.org/pipermail/freebsd-net/2011-April/028728.html says

NVIDIA bridge controller is known to have MSI issues for a long time in FreeBSD

While I don't entierly understand what this all means, the Macmini4,1 does have a Nvidia MCP89 chipset.

[probonopd]

Possibly @landonf knows how to get the Ethernet part of the BCM57765 in the Macmini4,1 to work properly on FreeBSD?

[probonopd]

https://wiki.freebsd.org/IntelMacMini says

To get bge(4) working, add the following to /boot/loader.conf:
hw.pci.enable_msi="0"

So it was working at some point?

[probonopd]

https://wiki.freebsd.org/AppleMacbook mentions under "Wired network seems to hang": hw.msk.msi_disable=1.

TODO: Test whether this helps.

[probonopd]

According to https://bsd-hardware.info/?id=pci:14e4-16b4-14e4-16b4 (at least) these devices use it:

• iMac12,1
• iMac12,2
• MacBookPro8,1
• MacBookPro8,2
• MacBookPro8,3
• MacBookPro9,1
• MacBookPro9,2
• Macmini4,1
• Macmini5,1
• Macmini5,2

[probonopd]

Submitted upstream:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260881

[probonopd]

A user with an Acer Aspire v3-571g (likely containing a BCM57785) is reporting a similar symptom here.

[probonopd]

Macmini4,1:

• Running hello-0.8.0_0H21-FreeBSD-13.1-amd64.iso (FreeBSD 13.1-BETA2)
• Boot without Ethernet plugged in
• Run Logs application
• Plug Ethernet in
• Last line: dhclient: New routers (bge0): 192.168.xxx.xxx - and shortly thereafter the system becomes unresponsive
• Repeat with hw.msk.msi_disable=1, does not help