Enable HYP mode on RPi-2 when booting

[yann-morin-1998]

Hello!

The Raspberry Pi 2 has a Cortex-A7. This processor has the virtualisation extensions:

Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm

Both ARM and Wikipedia agree that Cortex-A7 has hardware virtualisation:

https://en.wikipedia.org/wiki/ARM_Cortex-A7
http://www.arm.com/products/processors/cortex-a/cortex-a7.php
http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0464f/BABHJJBC.html

Unfortunately, virtualisation is not usable, as the bootloader disables HYP mode before booting the kernel:

CPU: All CPU(s) started in SVC mode.
[...]
kvm [1]: HYP mode not available

which makes it impossible to use virtualisation, even though the processor is supposedly capable.

Would it be possible to keep the CPU in HYP mode prior to booting the OS (linux or whatever), please?

Thanks! :-)

[yann-morin-1998]

Ping? ;-)

Would it be possible to at least get a status on this, even if it takes some time to solve?

Thanks! :-)

[popcornmix]

You can always use your own startup code. See:
http://www.raspberrypi.org/forums/viewtopic.php?f=72&t=100752
I'll look into leaving the arm in HYP mode.

[yann-morin-1998]

Oh, someone's already working on this! Great!

Thanks! :-)

[yann-morin-1998]

BTW, it is safe to leave the CPU in HYP mode, even when booting a non-virtualisation-aware Linux kernel, as the Linux kernel has historically always issued a switch to SVC at boot time. See this excellent LWN.net article: https://lwn.net/Articles/557132/ . Quoting:

As it turns out, it is reasonable to recommend that bootloaders always boot the kernel in HYP mode if it is available because even legacy kernels always make an explicit switch to SVC mode at boot time, even though they expect to boot into SVC mode already.

Regards.

[popcornmix]

If you have a patched version of the startup code linked in the forum that leaves the processor in HYP mode, I'd be happy to consider including it.

[mangelajo]

Looking forward to see the bootloader running the kernel in HYP mode.

[mestery]

++, I'm excited to see this land as well!

[mangelajo]

Do we have source code for the bootcode.bin ? , I have experience and I could look at this, even if I used RE to find and modify when the processor flags are changed, it seems that I couldn't redistribute the binary due to the license?, that's not cool

[popcornmix]

bootcode.bin is the GPU bootcode - nothing to do with the arm startup code.
The arm boot code we use is here: http://www.raspberrypi.org/forums/viewtopic.php?f=63&t=98367&start=375#p697474
and you can easily use kernel_old=1 and prepend your own bootcode to the kernel if you want to change it.

[Ruffio]

@mangelajo any progressive on this one? It looks like it is a wanted feature :-)

[TimJDFletcher]

I've written about a technique to enable HYP booting with u-boot on the Pi2 if that helps:
https://blog.night-shade.org.uk/2015/05/booting-a-raspberry-pi2-with-u-boot-and-hyp-enabled/

[yann-morin-1998]

@TimJDFletcher This is very interresting!

I indeed had a look to U-Boot, but I don't personally see the point of using such a "big" bootloader, when all I need is running a kernel. What is nice about U-Boot is that it already has support for booting various boards with HYP mode enabled, and the sole reason I looked at it: to see what they were doing that was missing on the RPi-2 (but I have to admit it is striding off limits to my understanding at some points! ;-) ).

What is very interesting about your post, however, is this rpi2-hyp-boot. That is very promising! Do you think this is something that may land into U-Boot itself at some point?

Thanks! :-)

Regards.

[TimJDFletcher]

rpi2-hyp-boot is separate to u-boot and can be used without u-boot just by glueing it on the front of the kernel image. I lifted it from https://github.com/slp/rpi2-hyp-boot documented here: http://blog.flexvdi.es//?p=139

My hope is that u-boot picks this up and includes in it the RPi boot loader, I can see that Raspbian might even want to use it. The down side to this is that QEMU on the Pi is finicky due to IRQ routing so this is likely to be a niche feature at best.

I have various embedded systems that use u-boot and find it useful as a boot loaded, especially as I have a serial console but no keyboard etc on my Pi.

[popcornmix]

How useful would it be to enable HYP mode (like https://github.com/slp/rpi2-hyp-boot/blob/master/rpi2-hyp-boot.S does) in the default boot code that runs before kernel starts?

I've have tested this before and it seems to cause no observable change in behaviour for linux.

@pelwell @notro any opinion on this?

[TimJDFletcher]

Lots of background from the LWM article linked above https://lwn.net/Articles/557132/

Basic answer is that since 3.6 the kernel is aware of HYP mode and will switch out of it automatically and transparently.

Another good selling point for the Pi2 is that it's the cheapest KVM capable ARM board by a fairly long way.

[pelwell]

It looks like a desirable feature, and if the standard kernel does indeed switch to SVC mode (which it appears to do here) then it should be transparent to most users.

[yann-morin-1998]

@popcornmix

How useful would it be to enable HYP mode [...] in the default boot code that runs before kernel starts?

Yes, that would be a good thing. As I already mentioned (and others have also confirmed), the kernel explicitly does a switch to SVC very early, and does not care to be started in HYP mode.

Of course, this requires at least a 3.6 kernel, but I would take for granted that the RPi2 will always boot such a kernel.

Thanks! ;-)

[mangelajo]

@popcornmix Sounds good, let's HYP! ;)

[popcornmix]

Can you test latest rpi-update firmware?
That entry.S startup code should leave the arm in HYP mode when it branches to kernel.img entrypoint.

[P33M]

[    0.053974] CPU0: thread -1, cpu 0, socket 15, mpidr 80000f00
[    0.054021] [bcm2709_smp_prepare_cpus] enter
[    0.054177] Setting up static identity map for 0x8240 - 0x8274
[    0.113786] [bcm2709_boot_secondary] cpu:1 failed to start (9420)
[    0.114180] [bcm2709_secondary_init] enter cpu:1
[    0.114236] CPU1: update cpu_capacity 1024
[    0.114245] CPU1: thread -1, cpu 1, socket 15, mpidr 80000f01
[    0.133761] [bcm2709_boot_secondary] cpu:2 failed to start (9420)
[    0.134107] [bcm2709_secondary_init] enter cpu:2
[    0.134139] CPU2: update cpu_capacity 1024
[    0.134147] CPU2: thread -1, cpu 2, socket 15, mpidr 80000f02
[    0.153805] [bcm2709_boot_secondary] cpu:3 failed to start (9420)
[    0.154060] [bcm2709_secondary_init] enter cpu:3
[    0.154090] CPU3: update cpu_capacity 1024
[    0.154098] CPU3: thread -1, cpu 3, socket 15, mpidr 80000f03
[    0.154189] Brought up 4 CPUs
[    0.154323] SMP: Total of 4 processors activated (153.60 BogoMIPS).
[    0.154365] CPU: All CPU(s) started in HYP mode.
[    0.154401] CPU: Virtualization extensions available.

Hypervisor seems OK, but bcm2709_boot_secondary is now reporting errors despite the CPUs coming online anyway?

Confirmed that this change is causing the error - here's from the last firmware push:

[    0.054157] CPU0: thread -1, cpu 0, socket 15, mpidr 80000f00
[    0.054203] [bcm2709_smp_prepare_cpus] enter
[    0.054369] Setting up static identity map for 0x8240 - 0x8274
[    0.113929] [bcm2709_boot_secondary] cpu:1 started (0) 17
[    0.114365] [bcm2709_secondary_init] enter cpu:1
[    0.114415] CPU1: update cpu_capacity 1024
[    0.114423] CPU1: thread -1, cpu 1, socket 15, mpidr 80000f01
[    0.133907] [bcm2709_boot_secondary] cpu:2 started (0) 18
[    0.134264] [bcm2709_secondary_init] enter cpu:2
[    0.134294] CPU2: update cpu_capacity 1024
[    0.134302] CPU2: thread -1, cpu 2, socket 15, mpidr 80000f02
[    0.153954] [bcm2709_boot_secondary] cpu:3 started (0) 16
[    0.154210] [bcm2709_secondary_init] enter cpu:3
[    0.154238] CPU3: update cpu_capacity 1024
[    0.154247] CPU3: thread -1, cpu 3, socket 15, mpidr 80000f03
[    0.154340] Brought up 4 CPUs
[    0.154475] SMP: Total of 4 processors activated (153.60 BogoMIPS).
[    0.154517] CPU: All CPU(s) started in SVC mode.

[popcornmix]

@P33M can you try with latest rpi-update code.

[TheSin-]

I just did and not sure if it's related but I updated yesterday things worked fine, updated today and now when I run chromium-browser I get

[29200:29263:1004/215143:ERROR:browser_gpu_channel_host_factory.cc(133)] Failed to create channel

Still trying to debug just wanted to report it incase you knew and it was related to this kernel update

[popcornmix]

@TheSin-
Not something I'm aware of. I doubt it's related to enabling HYP mode, but you can confirm.
HYP mode is enabled in firmware, so replace GPU firmware (/boot/start*.elf and /boot/fixup*.dat) from previous working update and see if that fixes it.
More likely it is an upstream issue in 4.1.10 (searching around may find users on other platforms seeing this).

[TheSin-]

okay I'll try both thanks, don't want to spam this thread, just been following it and noticed this issue directly after updating to the latest firmware git checkout. I'll use google for 4.1.10 and try the older elf/dat files see if it helps.

[TheSin-]

@popcornmix terribly embarrassed, somehow my config.txt and cmdline.txt got lost, please disregard everything I posted here, 4.1.10 and the new firmware both work fine for me now that I fixed that issue.

[swarren]

As far as I can tell from disassembling the latest VC-FW-provided ARM boot stub, the system now does boot in HYP mode. Can this bug be closed?

[yann-morin-1998]

Indeed, Linux now reports that all CPUs were booted inHYP mode:

[    0.027375] CPU: All CPU(s) started in HYP mode.
[    0.027383] CPU: Virtualization extensions available.

Closing this bug.

Thanks! :-)