maxcpus=1 stops Pi2 and Pi3 from booting

[pelwell]

As inadvertently discovered by @P33M, booting a Pi2 or Pi3 with maxcpus=1 on the kernel cmdline.txt stops it from booting with abnormally large timestamps in the kernel log and SD timeouts (presumably because the timer is running too fast). This smells like an issue with the secondary CPU startup and/or the local interrupt controller, but I haven't identified the specific cause yet.

[pelwell]

maxcpus=4, maxcpus= and an absent setting are the only options that work.

Note that a soft reboot is not enough to show the problem.

[popcornmix]

I've used maxcpus=1 a number of times, so it certainly has worked in the past.
It is currently working for me (I'm on 4.11 kernel)

pi@domnfs:~ $ uname -a
Linux domnfs 4.11.0-v7+ #2483 SMP Wed May 3 15:17:10 BST 2017 armv7l GNU/Linux
pi@domnfs:~ $ cat /proc/cmdline 
8250.nr_uarts=0 bcm2708_fb.fbwidth=1920 bcm2708_fb.fbheight=1080 bcm2708_fb.fbswap=1 vc_mem.mem_base=0x3dc00000 vc_mem.mem_size=0x3f000000  maxcpus=1 console=ttyS0,115200 kgdboc=ttyS0,115200 console=tty1 nfsroot=192.168.4.43:/home/dc4/rootfs ip=dhcp rootwait
pi@domnfs:~ $ cat /proc/cpuinfo 
processor	: 0
model name	: ARMv7 Processor rev 4 (v7l)
BogoMIPS	: 76.80
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

Hardware	: BCM2835
Revision	: 2a02080
Serial		: 00000000f45df3f2

[popcornmix]

I seem to get a mixture of successful and unsuccessful boots with maxcpus=1.
It seems to depend when the timer jump occurs and by how much whether it is fatal. E.g.

[   10.169931] Freeing unused kernel memory: 1024K
[  671.062266] INFO: rcu_sched self-detected stall on CPU
[  671.072361] 	0-...: (2 ticks this GP) idle=1bb/140000000000001/0 softirq=59/59 fqs=0 
[  671.082311] 	 (t=66077 jiffies g=-257 c=-258 q=2)
[  671.092181] rcu_sched kthread starved for 66077 jiffies! g4294967039 c4294967038 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1
[  671.102573] rcu_sched       S    0     8      2 0x00000000
[  671.112714] [<8074011c>] (__schedule) from [<8074079c>] (schedule+0x50/0xa8)
[  671.123014] [<8074079c>] (schedule) from [<80744060>] (schedule_timeout+0x1e8/0x33c)
[  671.133226] [<80744060>] (schedule_timeout) from [<80184cd4>] (rcu_gp_kthread+0x478/0x950)
[  671.143478] [<80184cd4>] (rcu_gp_kthread) from [<8013cfec>] (kthread+0x13c/0x16c)
[  671.153944] [<8013cfec>] (kthread) from [<80108148>] (ret_from_fork+0x14/0x2c)

[pelwell]

The mystery deepens. Pi2 and Pi3 both use the ARM architectural timers as their clock source, accessed from via the p15 co-processor interface by the arch_counter_get_cntpct function. Logging the values returned by this function shows a strange discontinuity around the time everything starts to go wrong:

[    4.919827]  mmcblk0: p1 p2
[  894.754884] arm_arch_timer: Timer history (400000029):
[  894.754884] arm_arch_timer:  5b54ec1
[  894.754880] arm_arch_timer:  5b55768
[  894.754878] arm_arch_timer:  5b55830
[  894.754882] arm_arch_timer:  5b560c4
[  894.754882] arm_arch_timer:  5b56179
[  894.754880] arm_arch_timer:  5b56a26
[  894.754881] arm_arch_timer:  5b56ae2
[  894.754882] arm_arch_timer:  400000020

And from later on:

[  894.754884] arm_arch_timer: Timer history (40000000e):
[  894.754881] arm_arch_timer:  400000019
[  894.754882] arm_arch_timer:  400000042
[  894.754878] arm_arch_timer:  400000055
[  894.754879] arm_arch_timer:  400000003
[  894.754879] arm_arch_timer:  40000000e
[  894.754881] arm_arch_timer:  400000062
[  894.754879] arm_arch_timer:  400000000
[  894.754879] arm_arch_timer:  4000000ab

Notice how the timestamps are now shuffling back and forth rather than monotonically increasing. It is as if this register has been switched into another mode, but I don't know what that mode would be and I can't see any evidence of such a switch; the frequency (CNTFRQ) remains the same (19.2MHz), as does the control register (CNTKCTL).

Are there any ARM gurus out there who can shed some light? @lategoodbye? @ED6E0F17?

[pelwell]

@swarren?

[swarren]

In the maxcpus!=4 case, some CPUs will permanently be executing the loop in the armstub waiting to be released. Hence, the content of this memory is permanently important. Is the kernel accidentally re-using that page for something else, thus causing the unused secondary CPUs to go off an execute random code? I don't know why that would always cause arch timer issues though.

When all 4 CPUs are in use, all the CPUs jump out of that code very early on, thus making it quite unlikely any such bug would be hit.

The value 400000020 looks special, as if it's a 400000000 flag plus value 20. Perhaps you can add code into writel and friends that checks for that value (or something in that range) and prints a backtrace in case it's caused by a stray register write?

[pelwell]

Ooh, interesting thought:

c0000000: 0000004d 00000000 00104278 000e822c M.......|xB..,...
c0000010: 00000000 0009dfe4 00000000 00000000 ........|........
c0000020: 0000004d 00000000 000f1664 00000000 M.......|d.......
c0000030: 00000000 0009de90 00000000 00000000 ........|........
c0000040: 0000004d 00000000 00104288 00000000 M.......|.B......
c0000050: 00000000 0009c494 00000000 00000000 ........|........
c0000060: 0000004d 00000000 00104294 00000000 M.......|.B......
c0000070: 00000000 0009d4a8 00000000 00000000 ........|........
c0000080: 0000004d 00170000 0010429c 00000000 M.......|.B......
c0000090: 00000000 0009d384 00000000 00000000 ........|........
c00000a0: 0000004d 00170000 001042a4 00000000 M.......|.B......
c00000b0: 00000000 0009d260 00000000 00000000 ....`...|........
c00000c0: 0000004d 00170000 001042b0 00000000 M.......|.B......
c00000d0: 00000000 0009d13c 00000000 00000000 ....<...|........
c00000e0: 0000004d 00170000 001042b8 00000000 M.......|.B......
c00000f0: 00000000 0009d01c 00000000 00000000 ........|........

The stub is loaded to 0xc0000000 (0x00000000 uncached, as seen from the VPU), and it has clearly been trashed by something.

Thanks for the suggestion - I'll follow up with an explanation when I find one.

[pelwell]

I have a fix which is somewhere between a solution and a workaround:

	reserved-memory {
		#address-cells = <1>;
		#size-cells = <1>;
		ranges = <>;

		rpi,stubs {
			reg = <0x0 0x1000>;
		};
	};

This bit of Device Tree magic will declare the first page as off-limits.

[lategoodbye]

@pelwell You were faster than me. I think this could be an acceptable way.

[lategoodbye]

Is the size of the armstub variable?

[swarren]

I wonder if the kernel code should reserve this itself; the memory can be released if it's known that all possible CPUs have been released from the spin loop. Let met check...

[swarren]

Looks like this is basically the correct approach, although the kernel docs recommend using /memreserve/ instead:

https://www.kernel.org/doc/Documentation/arm64/booting.txt

The boot loader is expected to enter the kernel on each CPU in the
following manner:

- The primary CPU must jump directly to the first instruction of the
  kernel image.  The device tree blob passed by this CPU must contain
  an 'enable-method' property for each cpu node.  The supported
  enable-methods are described below.

  It is expected that the bootloader will generate these device tree
  properties and insert them into the blob prior to kernel entry.

- CPUs with a "spin-table" enable-method must have a 'cpu-release-addr'
  property in their cpu node.  This property identifies a
  naturally-aligned 64-bit zero-initalised memory location.

  **These CPUs should spin outside of the kernel in a reserved area of
  memory (communicated to the kernel by a /memreserve/ region in the
  device tree)** polling their cpu-release-addr location, which must be
  contained in the reserved region.  A wfe instruction may be inserted
  to reduce the overhead of the busy-loop and a sev will be issued by
  the primary CPU.  When a read of the location pointed to by the
  cpu-release-addr returns a non-zero value, the CPU must jump to this
  value.  The value will be written as a single 64-bit little-endian
  value, so CPUs must convert the read value to their native endianness
  before jumping to it.

[popcornmix]

The armstub only uses 0x0-0xff.
ATAGS (when enabled) are at 0x100.

[lategoodbye]

According to the binding document and this example, we should add a "no-map" property to the stubs node ?

[pelwell]

Thanks, @swarren @lategoodbye, having had time to read around I think the /memreserve/ option is preferable. Interestingly, it is already conditionally enabled for 64-bit (bcmrpi3_defconfig) builds of bcm2710-rpi-3-b.dts, so I propose to move this to bcm283x.dtsi. Since bcm2835-based devices only have a single ARM core and tend to be more limited in RAM, I could make it conditional on CONFIG_SMP, but that seems unnecessary for the sake of 4KB.

[pelwell]

I have a patch adding /memreserve/ for all Pis waiting to go - I think it's safe to merge.

While we're in this area, I think it's time to add a wfe to the ARMv and ARMv8-32 stubs to reduce power consumption and bus traffic during startup (and later, if maxcpus=<n> has been used to keep some cores offline). The corresponding kernel-side change (adding an sev to the secondary-core-start function) is harmless, but a wfe with an sev will cause the secondary cores to sleep indefinitely.

@popcornmix - I propose merging both kernel side patches now, and I'll send you the armstubs patches - one for armstub7.S, and one for the generated header - for perusal and possible delaying.

[popcornmix]

@pelwell agree that adding the wfe/sev is desirable, but the wfe alone will break older kernels.
Adding the sev for now sounds good.
The wfe is trickier, but should be added in the future.

[popcornmix]

@pelwell agree that adding the wfe/sev is desirable, but the wfe alone will break older kernels.
Adding the sev for now sounds good.
The wfe is trickier, but should be added in the future.

[pelwell]

I've pushed the kernel-side commits.

[lategoodbye]

So these changes must go upstream or we are in trouble after release of the new armstubs?

[pelwell]

People using the upstream kernel can always put copies of the old stub files on their SD card/network boot directory, but that isn't ideal. I'll upstream these patches as well.

[swarren]

Having to copy old firmware files sounds like a nasty regression that can cause user-visible problems. I'd like to see this rolled out in a smoother fashion if at all possible. (1) Get the required sev present in all OSs (this includes RPi Foundation Linux, upstream Linux, Linux distro kernels, all non-Linux OSs that exist and use SMP), then wait a decent amount of time, and only then update the stubs in a way that requires an OS update. Alternatively, conditionalize the feature on some setup.txt flag that the RPi disk images can set, but nobody else does yet. The default value of the flag could be flipped in e.g. 6 months after people have time to convert.

[pelwell]

We're considering adding flags or a "breaking step" counter to our DTB that the firmware can check, which would then allow it to nop-out the wfe for old kernels. However, I have a feeling that wouldn't upstream since they can be very picky about what constitutes acceptable DT content.