ARM64 Only Run on CPU0

[caprice-j]

I run my RPI3 with custom 64-bit Kernel (cross-compiled on my laptop Ubuntu) in 64-bit ARM execution mode (i.e. arm_control=0x200 in /boot/config.txt) according to @Electron752 's experimental post:
https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=158127

It's running as aarch64 (confirmed by lscpu), but only on CPU0. I also checked by top result but only shows one CPU.

Other 3 CPUs are somehow failed during booting. dmesg says:

[    0.000000] Linux version 4.9.11-v8+ (pcu@ubt) (gcc version 6.2.1 20161016 (Linaro GCC 6.2-2016.11) ) #2 SMP PREEMPT Wed Feb 22 01:51:25 EST 2017
[    0.000000] Boot CPU: AArch64 Processor [410fd034]
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: UEFI not found.
[    0.000000] cma: Reserved 8 MiB at 0x000000003a800000
[    0.000000] On node 0 totalpages: 241664
[    0.000000]   DMA zone: 3776 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 241664 pages, LIFO batch:31
[    0.000000] /cpus/cpu@1: missing enable-method property
[    0.000000] /cpus/cpu@2: missing enable-method property
[    0.000000] /cpus/cpu@3: missing enable-method property

I googled for what "missing enable-method property" means, which I believe came from http://lxr.free-electrons.com/source/arch/arm64/kernel/cpu_ops.c#L85 but I have no idea what to do.

Multi-core execution is highly important for me since I'd like to run KVM on top of RPI3. Even without enabling KVM (i.e. original Electron752's bcmrpi3_defconfig), RPI3 only recognized the uni-processor. I'm hoping to use isolcpus=3 setting, but it's only possible in multi-core execution.

Does anyone have similar issues?

Other facts:

• I compiled my kernel on rpi-4.9.y branch (commit 204050d)
• my 64-bit kernel is placed in /boot/kernel8.img
• I can ssh to uni-processor RIP3
• when I executed sudo make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -j8 modules_install on my Ubuntu, the following messages appeared before installing modules:
  arch/arm64/Makefile:23: ld does not support --fix-cortex-a53-843419; kernel may be susceptible to erratum

Full output of dmesg:
https://gist.github.com/caprice-j/6f653ca3d7f6067f35a91e0e89388935

[Electron752]

Hi, glad to hear you find arm64 interesting!

The build and install process has changed since then which was about a year ago.

I think the issue is that you might not have copied the 64-bit version of bcm2710-rpi-3-b.dtb which should have been built by the build process. It should be located at arch/arm64/boot/dts/broadcom/bcm2710-rpi-3-b.dtb.

Also, you don't need arm_control=0x200 anymore. In fact, It doesn't works for me anymore.

To setup do:
cp arch/arm64/boot/Image /boot/kernel8.img
cp arch/arm64/boot/dts/broadcom/bcm2710-rpi-3-b.dtb /boot/
cp -r arch/arm64/boot/dts/overlays /boot/
And finally install the modules.

If things work you should not need any setting changed in config.txt using you are already pointing at a custom DT or kernel.

I don't believe KVM works at the moment. I've heard the RPI doesn't have the correct interrupt controller. I've been thinking about it some and how it might be possible to fake the interrupt controller or use a custom kernel in the VM. But I haven't put much effort into making it work.

[caprice-j]

@Electron752 So thanks for a quick reply! And so thank you for your great achievement. I'm really amazed when I saw the aarch64 string on my tiny single-board computer.

I'm out now but will try in a few hours according to your suggestion. Edit: Yes now my RPI3 is working with 4 cores. In fact my (old) bcm2710-rpi-3-b.dtb doesn't contain enable-method specificaition but the one under arch/arm64/boot/dts/broadcom/bcm2710-rpi-3-b.dtb has. I hope this will fix uniprocessor issue.

Besides that, it's sad to hear we don't have correct interrupt controller... I've just started to check about interrupt controller in RPI. It seems in RPI2 there was some attempts to enable VGIC: https://blog.night-shade.org.uk/2015/05/kvm-on-the-raspberry-pi2/
Original threads: https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=99284

I'm now thinking about implementing such a minimal (virtual) interrupt controller for KVM to work on RPI3, but I'm a kernel newbie and highly likely that I underestimate the hardness for implementing it. Do you happen to know some references how Linux Kernel implemented such a controller?

[Electron752]

I found some information in the Documentation directory of the kernel source, but like most of these documents it doesn't tell everything that's needed. I think its assumed that if you change the kernel, you can just read the source code.

Documentation/virtual/kvm

Part of the reason I haven't put any effort into this is that I'm not 100% sure these RPIs really have enough RAM in their current form to meaningfully run virtual machines and the RPI 3 is cheap enough that I can just buy another real RPI 3 if I need another machine. For software development, if I want to make something that isn't RPI specific(even some kernel changes) I just write it on a Intel PC in a virtual machine and recompile it for the RPI when I'm finished. I think a cheap VT enabled Intel PC can be bought for a few hundred dollars US(I'm a big fan of the Intel NUC). Finally, virtual machine time on providers like Amazon.com in the US which runs on Intel is rather cheap at the moment. It may go up in the future, but it's cheap now.

Trying to get KVM to work on the RPI 3 in ARM64 mode would be a cool learning experience though! I've been interested in the concept of virtual machines for quite some time now.

I'm curious about your thinking on the KVM concept on the RPI is? Are you thinking of web servers and such? Just interested in it for learning, or are you wanting to making something that actually gets used?

BTW, the erratum point is a whole issue in itself that's an architectural issue with ARM and MIPS that Intel/AMD doesn't need to deal with as much. They can just upload new microcode at boot time, which is something I don't think ARM and MIPS can easily do at present.

[caprice-j]

As you explained, it is true that the latest RP series as of Feb 2017 is not so powerful that can work as an alternative for a cloud web server. However, during the survey I noticed that during the transition from RPI2 to RPI3 the main memory size has been doubled, but the price remains the same. I'm super optimistic that the same thing would possibly happen when the transition happens from RPI3 to RPI4 (i.e. 2GB memory in the same price as current one). I think it plausible because we already have such a product (called PINE64) by $29. It has A53 CPUs and 2GB memory: https://www.pine64.org/?product_cat=boards

If the memory doubles, I think it's plausible to use RPI as an inexpensive alternative for research/educational cloud platform. A research group emulated cloud network architecture by connecting 50+ RPI: https://pdfs.semanticscholar.org/c8e4/7ef1ff46916994910fb10fcf583b8b044229.pdf

And for the further future, when RPI5 appears it might have 4GB memory by again the same price, 64bit addressing might become from for fun to necessary. At least I think it's worth to investigate whether there is any performance advantage for 64-bit Kernel on single computers.

The main reason I'm now thinking about KVM on RPI is such a cloud is useful for cloud architecture classes (I used to build a cloud by composing physical parts, but I think composing tens of raspberry pi thinking about physical network toplogy would be much more realistic and rewarding). Since cloud providers usually give users the virtualized environment, I think if KVM (preferably 2 or more machines in the future) works we can discuss such decisions in classes.

Buying another machine is another option and I think that technique will be combined as a hardware-based virtualization technology on such a cheap cloud.

Thanks for providing info for erratum.

[Electron752]

The PINE64 is cool. Very impressive.

It does use the same chip manufacture as the ODROID I bought a few months back, which from a performance point of view blows the RPI 3 out of the water. The only issue is that the software support isn't that good. I think the ODRIOD is stuck on some version 3 kernel which means many modern Linux distributions won't work well. I do know mainline Linux has some people working on getting it to work on newer kernels.

I have heard that it's going to be difficult for the RPI 4 to make the transition to 2GB with the current chipset since the GPU only uses 30 bit addresses. So to go higher and still keep the same GPU it's probably going to require 64 bit with fragmented RAM physical addresses. I've heard 2GB is about the most that can be realistically done on 32 bit arm regardless of the chipset.

Thanks for telling me about the Education/Research part of getting KVM to work. That does make things interesting and I think is in line with the spirit of what the RPI is all about.

As for 64 bit performance, from what I've seen myself is it's relatively performance neutral vs. 32 bit arm. Meaning it appears to run at about the same speed. Code size is very comparable as well. The main advantage at this point appears to be the ability to use more RAM, larger swap files, better encryption hardware instructions, and better security from features such as address space randomization.

[caprice-j]

• ODRIOD
  Interesting story. It seems justifying the focus on 32-bit but as latest Kernel as possible.

• GPU only uses 30 bit addresses
  That's an important issue. I'm ignorant of that...! Why are 32bit arm CPUs limited to 2GB? Is there some similar addressing issues?

• Education
  Yes, I think we students don't have much opportunities to learn about 64 bit architectures by hands-on, so if feasible that's great. I see 64-bit OpenSUSE on RPI3 as an event in the same scope.

• 64-bit performance
  Your comments are fair. I vaguely assumed since RPI series are used as media centers 64 bits capacity would make a difference in some graphic processing, but I tottaly forgot to think about GPU. The FAQ of Pine64 says 64-bit is much more power efficient but it doesn't seem to justify the development effort for transition to 64-bit.

[Electron752]

1. 32 bit arm is typically limited to 2GB due to the way virtual memory is setup. < 2GB user mode address and > 2GB kernel address. I think > 2GB could be made to work, but it sounds complex.

2. I need to do some more graphics benchmarks with 64 bit. Most of the benchmarks I did were things such as ffmpeg software video encoding and decoding and some old benchmarks such as bytecpu. I've heard in theory cpu bit bliting and memcpy can be faster with 64 bit since it can move 8 bytes at a time.

[caprice-j]

Interesting. I thought 32-bit limit is 4GB. I'm now curious of how PINE64 developers think about the problem.

Okay, then first I'll try the traditional benchmark would return the similar results. I need some research on 64-bit advantage for graphics.

Thank you for sharing your insights. Since my original problem is solved, I'll close this issue.

[caprice-j]

@Electron752 FYI: my first KVM-QEMU guest OS is running on RPI3. I'll check what cannot be executed by KVM in a few weeks

[Electron752]

Cool. Is this running a 64 bit host or a 32 bit host? RPI3 can run both ways.

[caprice-j]

64bit host. Oh no, I was wrong -- guest QEMU OS run but without KVM support. When I passed -enable-kvm, it returned qemu-system-arm: /home/pi/os/qemu/qemu-2.8.0/exec.c:611: cpu_address_space_init: Assertion asidx == 0 || !kvm_enabled()' failed.`. I need more investigation.

[caprice-j]

I thought cpuinfo shows some flags for virtualization support like Intel's vmx or svm (ARMv8 should support it), but I got:

pi@raspberrypi:~ $ lscpu
Architecture:          aarch64
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    1
Core(s) per socket:    4
Socket(s):             1
CPU max MHz:           1200.0000
CPU min MHz:           600.0000

pi@raspberrypi:~ $ cat proc/cpuinfo
... (cut) ...
processor       : 2
BogoMIPS        : 38.40
Features        : fp asimd evtstrm crc32
CPU implementer : 0x41   # ARM
CPU architecture: 8      #
CPU variant     : 0x0
CPU part        : 0xd03  # Cortex-A53
CPU revision    : 4      # minor revision
... (cut) ...

(comments after # are mine)

So it shows only four features. Compared to lots of features in 32-bit execution mode (https://www.raspberrypi.org/forums/viewtopic.php?f=28&t=138453), I'm worried that something wrong is happening on my CPU.

[caprice-j]

I found out the detailed procedures of 64-bit mode kernel: https://devsidestory.com/build-a-64-bit-kernel-for-your-raspberry-pi-3/

[Electron752]

Cool. Good to hear that 64 bit is getting some attention.

On a related topic, I have a pending patch that I sent to upstream to make vc04_services/vchiq work with a 32 bit OS like Raspbian when then kernel is compiled for 64 bit. This is like my 4th attempt at getting this in so I'm crossing my fingers this time. As you know, the foundation has no plans to convert the user mode part of Raspbian to 64 bit anytime in the near future.

http://lists.infradead.org/pipermail/linux-rpi-kernel/2017-March/005670.html

Except for the camera, their isn't really isn't that many RPI specific changes left that are still needed for a very functional 64 bit kernel. The camera still needs work though.

[caprice-j]

About the interrupt controller of RPI3, I was recommended this patch: https://lists.cs.columbia.edu/pipermail/kvmarm/2017-February/023469.html

[Electron752]

Thanks for the link. I might see what I can do with VMs in awhile.

Right now, I'm still struggling to get a fully functional ARM64 working. It looks like my V4 was rejected again, and I'll need to make a V5. Of the year or so I've been tinkering with this stuff, I would say only about 10%-20% of the time I put into this has been real actual development time. The rest has been trying to get things through the various processes.

The github based system is tolerable, but the upstream e-mail based system is very, very slow...

[caprice-j]

Sorry for no reply for a long time. I was occupied by other tasks.

I haven't sent patches to kernel communities, but guessing from your stories sending patches is a big hussle... I read your discussion in the lists.infradead.org and it seems like the next time will be a success (by writing documented version 5)?

By the way, this URL says RPI3 uses a proprietary interrupt controller which I heard for the first time in this thread. Do you happen to know something about that?
http://www.spinics.net/lists/kvm/msg144642.html

[Electron752]

It appears the V5 was accepted and I was able to get the camera to work in arm 64 as well. Both sets of changes are waiting in the various staging areas. So as far as upstream kernel is concerned, arm64 should essentially be work complete.

Since I'm not associated with either the PI Foundation or Broadcom, I don't get any information that anybody else wouldn't. The only information I have on the interrupt controller is what in the public source code on github and in the mainline kernel. I have also collected a few of the public documents that are floating around. The most useful of which is titled, "BCM2835 ARM Peripherals".

Sometimes people go to the bare metal forum on the www.raspberrypi.org site and say your writing a bare metal app. Some times you can get information that way and sometimes people get kicked out. The PI foundation does have some pinned topics on where to get public information in that area. That's a good place to start.

[JamesH65]

You only get kicked off the forums if you do something wrong! The bare metal section is a good place to start for this stuff.

…

On 15 March 2017 at 06:32, Electron752 ***@***.***> wrote: It appears the V5 was accepted and I was able to get the camera to work in arm 64 as well. Both sets of changes are waiting in the various staging areas. So as far as upstream kernel is concerned, arm64 should essentially be work complete. Since I'm not associated with either the PI Foundation or Broadcom, I don't get any information that anybody else wouldn't. The only information I have on the interrupt controller is what in the public source code on github and in the mainline kernel. I have also collected a few of the public documents that are floating around. The most useful of which is titled, "BCM2835 ARM Peripherals". Sometimes people go to the bare metal forum on the www.raspberrypi.org site and say your writing a bare metal app. Some times you can get information that way and sometimes people get kicked out. The PI foundation does have some pinned topics on where to get public information in that area. That's a good place to start. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_raspberrypi_linux_issues_1868-23issuecomment-2D286653316&d=DwMFaQ&c=DpyQ_ftY536pf7wCBQXXU58xADDRY77THQzJu1OmzOo&r=w09_2ePv8G3zRjoV19Wm1Q6rI7CDlOns4PuRv2hHkek&m=J5ok4zUA4hMBDOs5CB57K3IS0TsfsT0Xv1GKOu9vW8w&s=9rLT1e6lLMi784q94UGhbuIM0XWMV7IZpdAG4OffGEI&e=>, or mute the thread <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ADqrHVqJfsVyESIgWIOU-2Dm5P-2D38eETi3ks5rl4YYgaJpZM4MP7gh&d=DwMFaQ&c=DpyQ_ftY536pf7wCBQXXU58xADDRY77THQzJu1OmzOo&r=w09_2ePv8G3zRjoV19Wm1Q6rI7CDlOns4PuRv2hHkek&m=J5ok4zUA4hMBDOs5CB57K3IS0TsfsT0Xv1GKOu9vW8w&s=8M8XN4oYRCKI4Bfq9tBNRMZfRx-KFaivVJHcnacu11M&e=> .

-- James Hughes Principal Software Engineer, Raspberry Pi (Trading) Ltd

[caprice-j]

Thank you for your suggestion and congratulations for your patch accept. It seems I don't even need to ask -- this post says RPI3 have a proprietary GIC: https://www.raspberrypi.org/forums/viewtopic.php?f=62&t=156639&p=1057546&hilit=GIC#p1057546

Okay, then I need to find userspace GIC support KVM and QEMU patches.

[caprice-j]

Applied QEMU/KVM patches but still got an error. I asked here and am waiting for a reply: https://lists.cs.columbia.edu/pipermail/kvmarm/2017-March/024111.html

By the way, I tried 64-bit Fedora 25 image on RPI3 distributed here: https://www.kraxel.org/blog/2017/02/fedora-25-images-for-qemu-and-raspberry-pi-3-uploaded/

And I confirmed taskset -c 3-3 qemu-system-aarch64 --enable-kvm -cpu host -machine type=virt -nographic -smp 1 -m 256 -kernel ~/os/aarch64-linux-3.15rc2-buildroot.img --append "console=ttyAMA0" returned very fast (compared with no --enable-kvm flag) bootup messages until endlessly halting after starting network... message (which correctly finished and led to login prompt with no --enable-kvm flag). So my configuration (which resulted in kvm accelerator not found error) is lacking of something Fedora correctly set up. As Raspibian OS, Fedora also does not have kvm-ok command, no virt-related flag in /proc/cpuinfo, but has /dev/kvm. I generated Fedora's /proc/config.gz and will investigate it tomorrow.

the buildroot.img is retrieved from: https://www.bennee.com/~alex/blog/2014/05/09/running-linux-in-qemus-aarch64-system-emulation-mode/

[caprice-j]

Other error on Fedora by userspace-emulated GIC QEMU build:

[root@localhost aarch64-softmmu]# taskset -c 3-3 ./qemu-system-aarch64 --enable-kvm \
-machine virt,kernel-irqchip=off -cpu host -machine type=virt -nographic -smp 1 \
-m 256 -kernel ~/os/aarch64-linux-3.15rc2-buildroot.img  \
--append "console=ttyAMA0" -net none

qemu-system-aarch64: KVM with user space irqchip only works when the host kernel supports KVM_CAP_ARM_TIMER

[caprice-j]

Surprisingly enough, I found out an article claiming OpenSUSE Leap 42.2 officially supports KVM on Raspberry Pi 3: https://news.opensuse.org/2016/12/05/opensuse-leap-42-2-gets-64-bit-raspberry-image/

It turned out that one of the SUSE developers is the one who sent userland-GIC emulation to QEMU, so I guess he incorporated necessary KVM patches in SUSE kernel. When I used his patched QEMU build (agraf/qemu no-kvm-irqchip branch), I successfully run my 64-bit Ubuntu by the following command:

./qemu-system-aarch64 --enable-kvm -m 256 -cpu host -nographic -machine virt,kernel_irqchip=off \
 -kernel vmlinuz-4.4.0-66-generic \
 -append 'root=/dev/vda1 rw rootwait mem=256M console=ttyS0 \
  console=ttyAMA0,38400n8 init=/usr/lib/cloud-init/uncloud-init \
  ds=nocloud ubuntu-pass=upass' \
 -drive if=none,id=image,file=xenial-server-cloudimg-arm64-disk1.img \
 -initrd initrd.img-4.4.0-66-generic \
 -device virtio-blk-device,drive=image \
 -netdev user,id=user0 \
 -device virtio-net-device,netdev=user0

(I extracted init.rd and vmlinuz from the above .img file by qemu-nbd)

I run a few numerical performance tests and confirmed KVM/QEMU MFLOPS rank with bare-metal ones.

[Electron752]

Like I said, at the moment I'm not really interested in KVM because I'm not sure these things have enough RAM at this time.

I especially like the comment in the link you sent me about some monitors not working with VC4 on RPI 3 for mainline. Until this morning VC4 didn't work at all on mainline on the RPI 3 period through HDMI. And the patches still haven't been merged by the staging maintainer in mainline so VC4 on RPI 3 is simply "vaperware."

[caprice-j]

I see. I'm wondering why you are so interested in 64-bit execution. My reason is that in 64-bit execution QEMU can use --enable-kvm for both 32-bit and 64-bit guest OSes (fast).

[Electron752]

This all got started because a bunch of people on the raspberrypi.org forum were noticing that the RPI 3 is 64 bit, yet no 64 bit kernel was available. So a few people on the raspberrypi.org forum were trying to get 64 bit bootstrapped. They had an issue with DMA on USB, so I thought I maybe could take a crack at it. USB is something I've been tinkering with on small microcontrollers with the PIC controllers for a long time now.

The whole thing was meant to be a community driven effort( of which I am part of the community and not paid in any way) but it quickly turned into one person making most of the contributions(me). At first I mostly contributed to small patches in downstream, but found out they(PI Foundation) want most of the effort to go to upstream. Someone who is involved in upstream invited me to send in a few patches and that's how upstream got started.

Honestly, it's just that I've put so much effort and time into this whole quest that it's hard to just let go. I'm thinking maybe some of the drivers and changes I've gotten into upstream may turn into something more. And yes, it would be good if other got involved.

Anyway, the whole 64 bit thing is low priority for people at this time and doesn't have much end user value. Yes, in some cases it may be faster and some cases it may be slower. But perhaps something will come out of it someday and people will find it useful.

[caprice-j]

I see. I'm also the one who thought official OS support 64-bit execution. I was excited when I saw your article about 64-bit make commands.

It sounds interesting you started from downstream and moved (and was encouraged ) to upstream. I guess reviewing codes would take lots of time for them.

OS classes I took at my university did not focus on device handlings like ioctl, and everything you're concerning looks new for me.

[Electron752]

OK, I really don't know where this is going. You asked for help and such, and I gave it. Since the issue is closed, if you have other questions perhaps you should open a new topic.

[caprice-j]

Sure. Thank you for your sincere help. I'll end this issue page here.