USB boot does not work on every RPI3

[JoramQ]

I've got two RPI3 v1.2 boards here. Made an SD card to set the OTP bit on both devices and also verified that is was set. On one board the RPI boots correctly and runs good as far as I've tested today. On the other RPI, with exactly the same USB drive(just moved it over to the other device, not changing anything) it does not boot correctly.
Half of the times the display stays black, other half of the time I get the squared colored test image. Even though i disabled this image and it does not show on the other RPI. Also the image is the only thing I see, no network connection, no commandline, just the image.

Both PCB's are the same, Model B V1.2. There is only a small difference in some numbers on the board. The one that is working has the QR code on the bottom is 24/06 and the code KCE MC1 1618 94V-OF1 I5. The one that is not working has the QR code on the bottom is 23/03 and the code KCE MC1 1608 94V-OF1 E4.

[ghollingworth]

Does it work reliably if you boot from an sdcard containing just bootcode.bin?

This should still boot from the USB drive, but uses the bootcode in bootcode.bin which has a couple of bugs fixed in it.

Gordon

[xmob]

I'm seeing pretty much the exact same issues.

2 x RPI3 (Both are KCE MC1 1610)

Initially, I was experimenting with PXE booting. First Pi worked exactly as expected, netbooting etc. The second Pi does not. Without an SD card the network never establishes a link and so doesn't do a DHCPDISCOVER. With an SD card containing bootcode.bin the Pi obtains a DHCP lease and makes the first request from the TFTP server. The client (Pi) then sends an abort response when the first block of start.elf is sent. The green LED flashes 4 times at this point.

Inspired by this issue I tested with booting from USB media. The first Pi boots a Raspbian Stretch image okay. The second does not, including when using an SD card with bootcode.bin.

I'm doing all of this headless, so I don't know what is present on the framebuffer. If it will help, I can see if there is anything on the serial console.

[patsteel]

Same issue. Please advise. Tried about everything at this point. Boot w/o SD card gets me just the red led. Booting with SD card with bootcode.bin and timeout file gets me four green flashes. Trying to boot from a 32GB Transcend SSD in a usb 2.0 case, which I have tried imaging in both Linux and Windows so pretty sure this is not the issue. Drive shows up just fine if I boot up from an SD card image.

[ghollingworth]

bootcode.bin.gz

Can you try the attached bootcode.bin on it's own, create an empty file called 'UART' alongside it. Then connect serial to the standard pins (GPIO 14, 15 on pins 8 and 10) serial 115200-8-n-1

Then power it up and report with the log output? This will give useful output for both USB MSD boot and network boot

Thanks

Gordon

See the Wiki for more information on this debug...

[xmob]

Thanks for helping with this.

The UART output when booting with this bootcode.bin is:

Raspberry Pi Bootcode

Found SD card, config.txt = 0, start.elf = 0, recovery.elf = 0, timeout = 0
Trying USB
Hub device found at addr 4, enumerating HUB
Initialise hub
Found 5 ports, multi_tt = 1
Setting interface 0
Enabling PORT POWER on port 1
Enabling PORT POWER on port 2
Enabling PORT POWER on port 3
Enabling PORT POWER on port 4
Enabling PORT POWER on port 5
Waiting for devices to respond to reset
Found device on port 1
Found highspeed device
Device found: type = Ethernet adapter, addr = 5
Trying booting from Ethernet device addr 5
Initialise ethernet with MAC b8:27:eb:cb:e2:62
Wait for Link up
Link up
Sending DHCP request
Waiting for dhcp_reply
Done ARP for 1.0.22.10 got 00:19:99:cf:bc:39




Raspberry Pi Bootcode

The second Raspberry Pi Bootcode appeared at around about the same time that the TFTP server logged:

Nov 13 19:54:45 nspawn atftpd[657]: Serving 74cbe262/start.elf to 10.11.12.14:49153
Nov 13 19:54:45 nspawn atftpd[657]: tsize option -> 2868132
Nov 13 19:54:45 nspawn atftpd[657]: Aborting transfer
Nov 13 19:54:45 nspawn atftpd[657]: Server thread exiting

I hope this provides some useful pointers. Of course, I'm more than happy to help in any way that I can.

[ghollingworth]

It looks like you need a little more debug output. Also can you run tcpdump on it as well...

sudo tcpdump ether host b8:27:eb:cb:e2:62 or ether host ff:ff:ff:ff:ff:ff

That should do it, it'll capture all broadcast packets as well and any either sourced or destined for the Raspberry Pi.

[xmob]

Log files attached below.

I think it should be pointed out that this process only attempts to netboot once in every three or so attempts. Most other times, the network link doesn't come up and the following can be seen on the UART output:

Found SD card, config.txt = 0, start.elf = 0, recovery.elf = 0, timeout = 0
Trying USB
Hub device found at addr 4, enumerating HUB
Initialise hub
Found 5 ports, multi_tt = 1
Setting interface 0
Enabling PORT POWER on port 1
Enabling PORT POWER on port 2
Enabling PORT POWER on port 3
Enabling PORT POWER on port 4
Enabling PORT POWER on port 5
Waiting for devices to respond to reset
Found device on port 1
Found highspeed device
Device found: type = Ethernet adapter, addr = 5
Trying booting from Ethernet device addr 5
Initialise ethernet with MAC b8:27:eb:cb:e2:62
Wait for Link up
Failed to initialise link
Trying booting from Ethernet device addr 5
Trying booting from Ethernet device addr 5
Trying booting from Ethernet device addr 5
Trying booting from Ethernet device addr 5
Trying booting from Ethernet device addr 5
Trying booting from Ethernet device addr 5

Even more occasionally, it is as seen above in my previous comment.

For completeness, the config.txt is:

# cat /srv/tftp/74cbe262/config.txt | grep -v '^$\|^\s*\#'
dtparam=audio=on

dhcp_tcpdump.log
tftpd.log
tftp_tcpdump.log
uart.log

[ghollingworth]

There's a 5 second delay waiting for the link to come up... Are you seeing it sitting at 'Wait for Link up' for five seconds? Just wondering if this is a problem with timing setup...

Gordon

[ghollingworth]

Try again with this, added some debug output while sending TFTP_RRQ and ACK

Note, this contains a lot of debug, since it will output debug for every TFTP packet, I'm guessing this will slow things down significantly but should give us an idea of why it's going wrong

bootcode.zip

[xmob]

Yep, seeing the 5 second delay.

Attached is the UART output.

At the moment, I'm testing this on a live network with separate DHCP, TFTP and NFS servers. This worked okay for me with the first Pi I setup. I'll allocate some time to setup up an isolated network with DHCP, TFTP and NFS all on the same server. This will make capturing traces much easier. Ideally, I'd like this to be ISC DHCP and atftpd as that's what I'm using for my current PXE environment. If you think it beneficial, I can build a dnsmasq environment as described in the tutorials.

uart_20171115.log

[ghollingworth]

It would be useful if you could tcpdump the packets as well as get the UART output, because this should show me what happens on that last packet...

Gordon

[JohnOH]

I hope I am not waking up a dead thread or hijacking. I have a similar problem but trying to boot from a USB WD PiDrive 1TB. I have made use of the debug bootcode.bin & UART to obtain output suggesting to me that the disk isn't spinning up quickly enough, see enclose UART log.
RasPi_SpecailBoot.txt

[ghollingworth]

Don't know about spinning up quickly enough, but it's not a problem with the disk not attaching in time (this is what the standard timeout is for).

It looks like it's failing to reply to the TEST_UNIT_READY command, not sure why this would be... Can you plug it into a PC and see it it subsequently works? I wonder whether the drive behaves slightly differently in some cases (i.e. it is doing some housekeeping that it needs to finish before starting up...)

Gordon

[JohnOH]

Many thanks for getting back to me so soon Gordon. As is often the case my further throughts have pointed the finger of blame towards me. I used this to build my HD and now realise that it doesn't create the FAT32 as partition 1 on the HD. Since a boot directory was copied to sda1 I presumed that it would accessed by bootcode.bin from the SDCard. I will rebuild with what I suspect is the correct two partition layout for the HD.

[JohnOH]

Pity, no change in the UART output after the change.
Command (m for help): p
Disk /dev/sdb: 931.5 GiB, 1000170586112 bytes, 1953458176 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x0b44bef7

Device Boot Start End Sectors Size Id Type
/dev/sdb1 * 2048 33556479 33554432 16G b W95 FAT32
/dev/sdb2 33556480 167774207 134217728 64G 83 Linux
/dev/sdb3 1639974912 1953458175 313483264 149.5G 83 Linux

[JohnOH]

Can you plug it into a PC and see it it subsequently works?

Now that the FAT32 is added as partition 1 the drive connects fine to my Mac. It seem to be quite sluggish before it is actually mounted on the Mac.

[ghollingworth]

I've added a little debug to this bootcode.bin, can you put it onto the MSD to get the UART debug out?

bootcode.zip

[JohnOH]

Here is the latest UART output.
UART2.txt

[JohnOH]

I tried bouncing the power to the RasPi while keeping the disk powered and it boots:
UART3.txt

[ghollingworth]

bootcode.zip

This just has the timeout increased to 20 seconds! Lets see what happens!

Gordon

[JohnOH]

Yo! we have action:
UART4.txt

[pelwell]

Just as a matter of interest, which power supply do you have for the PiDrive? The original supply was rated at 2A, but WD introduced a 2.5A supply intended for use with the Pi 3.

[JohnOH]

My PSU claims to be 3A, branded Ktec. It came along with the 1TB disk.

[ghollingworth]

I assume that this isn't going to help with booting directly from the hard drive though, since we cannot change the bootrom (that's fixed in the silicon), so it does mean you're going to have to have an SD card plugged in to get it to boot!

Will commit this change to bootcode.bin so you can at least do that....

[JohnOH]

Thanks for your help Gordon. I had already decided not to set the OTP bit for USB boot as I would like to retain the flexibility. Loading bootcode.bin from an easy to replicate SD card and then bringing in the HD is fine for me and applicable across the whole RasPi family I am expecting.

[JohnOH]

@ghollingworth I wonder if something could also be done about the bootcode.bin apparently locking into an apparent ethernet boot loop if the USB drive doesn't respond?

[ghollingworth]

Um...

Not sure what you'd want to do in this situation? If the USB drive doesn't respond or no suitable firmware is found on the device, what would you like it to do?

Gordon

[JohnOH]

I would hope it could restart from the beginning again checking all its options once more.

[JohnOH]

Perhaps also power the USB ports down and then up again?

[c64emulator]

Not sure what you'd want to do in this situation? If the USB drive doesn't respond or no suitable firmware is found on the device, what would you like it to do?

Some sort of signaling by LED (must) and maybe (nice) on video output or audio output?

[JohnOH]

@ghollingworth I saw your post on #838 and recent fixes to SDcard handling. Is there a version of bootcode.bin that includes the SDcard fixes and the 20 seconds timeout that you supplied on 5th March?

[ghollingworth]

I've created an internal pull request to change this timeout to 20 seconds... Should appear in the latest firmware (available through rpi-update) over the next couple of days.

[JohnOH]

Is there a link to the pull for this update?

[ghollingworth]

It'll be in the next release...

[popcornmix]

Latest rpi-update firmware contains this fix.

[JohnOH]

Loaded now, many thanks.

[JohnOH]

I wonder if this bootcode.bin could be network loaded and the remainder of the boot continue from a connected HDD.

[ghollingworth]

If the bootcode.bin is retrieved from the network then it'll require the rest of the firmware files to be located there (start.elf, fixup.dat, kernel7.img, config.txt etc.) but you can then put a suitable cmdline.txt to use the USB MSD for the ext4 filesystem.

Of course the ethernet booting has its own set of issues (much as anything)!

Gordon

[JamesH65]

I believe this can now be closed?