RPI Zero 2W incorrect memory size #5975

gugolple · 2024-02-20T14:42:38Z

Describe the bug
After doing an rpi-update on my raspberry pi zero 2W, the available ram reduced from ~484MB to 400MB. I do understand that the memory split will take an amount of ram (check logs section). I do not understand the reason of the change, but having less resources available in such a constrained board is not beneficial.

To reproduce
Executing the command sudo rpi-update.

Expected behaviour
Available ram for the system to be at around ~484MB.

Actual behaviour
Available ram for the system is found to be at around ~400MB.

System

$ cat /etc/rpi-issue
Raspberry Pi reference 2023-10-10
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 962bf483c8f326405794827cce8c0313fd5880a8, stage4

$ vcgencmd version
Dec  8 2023 12:42:28 
Copyright (c) 2012 Broadcom
version e02d33b3122450accf9dea471a177d3b5623f5ad (clean) (release) (start_cd)

$ uname -a
Linux raspberrypi 6.6.16-v8+ raspberrypi/firmware#1728 SMP PREEMPT Tue Feb  6 17:21:42 GMT 2024 aarch64 GNU/Linux

Logs
Reduced to only provide relevant information.

$ dmesg | egrep 'Memory|CMA|cma' --color -C3
[    0.000000] Machine model: Raspberry Pi Zero 2 W Rev 1.0
[    0.000000] Memory: 125944K/507904K available (13376K kernel code, 2208K rwdata, 4308K rodata, 4928K init, 1083K bss, 119816K reserved, 262144K cma-reserved)

$ cat /proc/meminfo
MemTotal:         409760 kB

$ vcgencmd get_mem gpu
gpu=16M

Additional context
I do not know if they are relevant, but I have attached the boot configurations just in case I have enabled a device that also is reserving ram at boot.

$ cat /boot/firmware/cmdline.txt 
console=serial0,115200 console=tty1 root=PARTUUID=e668d3db-02 rootfstype=ext4 fsck.repair=yes rootwait quiet splash plymouth.ignore-serial-consoles cfg80211.ieee80211_regdom=BE

$ cat /boot/firmware/config.txt | grep -v '#' | grep -v -e '^$'
dtparam=spi=on
dtparam=audio=on
camera_auto_detect=0
display_auto_detect=1
auto_initramfs=1
dtoverlay=vc4-kms-v3d,spi1-3cs
max_framebuffers=1
disable_fw_kms_setup=1
arm_64bit=1
disable_overscan=1
arm_boost=1
[cm4]
otg_mode=1
[all]
start_x=0
gpu_mem=16
enable_uart=1

Thank you.

The text was updated successfully, but these errors were encountered:

gugolple · 2024-02-20T14:52:47Z

Here are the logs of executing the same commands on my Raspberry Pi Zero W. These are coherent with a device with ~500MB of ram with ~64MB separated for the GPU split.

$ dmesg | egrep 'Memory|CMA|cma' --color -C3
[    0.000000] Memory: 374380K/458752K available (8694K kernel code, 1324K rwdata, 2828K rodata, 424K init, 836K bss, 18836K reserved, 65536K cma-reserved)

$ cat /proc/meminfo
MemTotal:         440340 kB

$ vcgencmd get_mem gpu
gpu=64M

popcornmix · 2024-02-20T14:55:16Z

Can you identify the exact update which caused this. See:
https://github.com/raspberrypi/rpi-firmware/commits/master

If you click on each commit the end of the url contains a git hash. Run
sudo rpi-update <hash>
to revert back to that version. Report the first version with this error.

pelwell · 2024-02-20T15:02:40Z

Some before and after comparisons would be useful - that looks to be all after.

I note that you've opted into the 64-bit kernel (and perhaps even the 64-bit OS - it's hard to tell). This is not recommended on a Zero 2 W because it will eat into your limited RAM.

gugolple · 2024-02-20T16:39:38Z

Can you identify the exact update which caused this. See: https://github.com/raspberrypi/rpi-firmware/commits/master

The issue was introduced at the commit e632362b0399b4ce331aacd9386685bc60938ab7

Some before and after comparisons would be useful - that looks to be all after.

I note that you've opted into the 64-bit kernel (and perhaps even the 64-bit OS - it's hard to tell). This is not recommended on a Zero 2 W because it will eat into your limited RAM.

Free before the commit (using commit af3ab905d42829d4ce860f19a0e049c3bcdbe35f):

$ free -h
               total        used        free      shared  buff/cache   available
Mem:           466Mi       165Mi       138Mi       1.2Mi       218Mi       300Mi

Free after appling the commit e632362b0399b4ce331aacd9386685bc60938ab7:

$ free -h
               total        used        free      shared  buff/cache   available
Mem:           400Mi       154Mi        88Mi       1.2Mi       209Mi       246Mi

gugolple · 2024-02-20T16:44:10Z

Some before and after comparisons would be useful - that looks to be all after.

I note that you've opted into the 64-bit kernel (and perhaps even the 64-bit OS - it's hard to tell). This is not recommended on a Zero 2 W because it will eat into your limited RAM.

Indeed I am running Raspberry Pi OS 64 bit version, but I do not see how it affects the topic at hand.

popcornmix · 2024-02-20T16:45:06Z

Ah - so was this the kernel update rather than the firmware?
That commit did move from 6.1 kernel to 6.6.

You should be able to confirm this by copying /boot/firmware/{start*.elf,fixup*.dat} from the "good" rpi-update commit and the "bad" commit somewhere safe, and check if just swapping those files over changes free memory available.

You can check firmware and kernel versions after a reboot with:

uname -a
vcgencmd version

pelwell · 2024-02-20T16:47:29Z

but I do not see how it affects the topic at hand.

Do you care about how much usable RAM your Z2W has?

pelwell · 2024-02-20T16:54:13Z

Note that as well as having unavoidably larger page tables, lower code density, etc., there is a common 64-bit kernel that runs on Pi 3s, 4s, 5 and Z2W. This means that your Z2W has used some of its RAM to hold support for booting a Pi 5 to the point where it can access its SD card, USB, and Ethernet interfaces. kernel7.img is only for Pi 3s and Z2W, therefore it is much smaller. Note also that the 64-bit kernels are GZIP compressed - uncompressed they are roughly 25MB.

gugolple · 2024-02-20T17:11:26Z

Ah - so was this the kernel update rather than the firmware? That commit did move from 6.1 kernel to 6.6.

You should be able to confirm this by copying /boot/firmware/{start*.elf,fixup*.dat} from the "good" rpi-update commit and the "bad" commit somewhere safe, and check if just swapping those files over changes free memory available.

You can check firmware and kernel versions after a reboot with:
uname -a
vcgencmd version

I have run out of time for the moment, but copying either set of files from the "bad" commit to the "good" one does not appear to solve the issue. I am missing to test the other permutation.

gugolple · 2024-02-20T17:12:02Z

Note that as well as having unavoidably larger page tables, lower code density, etc., there is a common 64-bit kernel that runs on Pi 3s, 4s, 5 and Z2W. This means that your Z2W has used some of its RAM to hold support for booting a Pi 5 to the point where it can access its SD card, USB, and Ethernet interfaces. kernel7.img is only for Pi 3s and Z2W, therefore it is much smaller. Note also that the 64-bit kernels are GZIP compressed - uncompressed they are roughly 25MB.

Correct, that affects memory utilization, not the amount of memory available to the system, that is the issue I am suffering.

popcornmix · 2024-02-20T17:21:16Z

I have run out of time for the moment, but copying either set of files from the "bad" commit to the "good" one does not appear to solve the issue. I am missing to test the other permutation.

I'd expect copying firmware from "bad" to "good" would either have no effect (i.e. it's not a firmware issue) or to make memory available worsen (i.e. it is a firmware issue). Can you confirm which result you had ("solving" wasn't an expected outcome from this test).

gugolple · 2024-02-20T20:36:20Z

The results from the improvised experiment:

	af3ab9	e63236
start.elf af3ab9 fixup.dat af3ab9	Correct	Bad
start.elf e63236 fixup.dat af3ab9	Correct	Bad
start.elf af3ab9 fixup.dat e63236	Correct	Does not boot
start.elf e63236 fixup.dat e63236	Correct	Bad

After seeing the results, I do believe the issue is with the new 6.6 kernel and not with the firmware.
Thank you so much @popcornmix for the support given. I believe the issue should be closed here and open in the kernel repo. Would you kindly confirm?

popcornmix · 2024-02-20T20:48:27Z

start*.elf and fixup*.dat should always be of the same version, so not booting is reasonable.

I've transferred this issue to the linux repo as that appears to be the cause of the increased memory.

As @pelwell says, it is worth testing how much of the increased memory is due to 64-bit kernel (which now also supports pi5, which the 32-bit kernel does not).

Is your OS 32-bit or 64-bit? If it's 32-bit then booting without arm_64bit=1 would be informative.
If it's 64-bit then it's trickier to test (will need a 32-bit install).

gugolple · 2024-02-20T20:54:28Z

As @pelwell says, it is worth testing how much of the increased memory is due to 64-bit kernel (which now also supports pi5, which the 32-bit kernel does not).

Is your OS 32-bit or 64-bit? If it's 32-bit then booting without arm_64bit=1 would be informative. If it's 64-bit then it's trickier to test (will need a 32-bit install).

Currently I'm running "Raspberry Pi OS (64-bit) : Raspberry Pi OS Lite".

What tests and versions would be intriguing to execute? I am open for suggestions.

pelwell · 2024-02-21T11:39:27Z

Although I've never quite got 460M for total mem, my results are close, seeing a ~60M drop when switching to the 6.6 kernel - whether by rpi-update or building my own.

Looking at the start of the /proc/iomem output with some annotations is instructive.

Before:

MemTotal:         440568 kB
00000000-1effffff : System RAM [-0.0M -> 496.0M]
  00000000-00000fff : reserved [-0.0M -> 496.0M]
  00210000-0121ffff : Kernel code [-16.1M -> 479.9M]
  01220000-0165ffff : reserved [-4.2M -> 475.7M]
  01660000-0199ffff : Kernel data [-3.2M -> 472.4M]
  0b200000-0ddfffff : reserved [-44.0M -> 428.4M]
  0de32000-0def2fff : reserved [-0.8M -> 427.7M]
  0def3000-0df66fff : reserved [-0.5M -> 427.2M]
  0df69000-0df69fff : reserved [-0.0M -> 427.2M]
  0df6a000-0df6afff : reserved [-0.0M -> 427.2M]
  0df6b000-1e08efff : reserved [-257.1M -> 170.1M]
  1e08f000-1e096fff : reserved [-0.0M -> 170.0M]
  1e097000-1effffff : reserved [-15.4M -> 154.6M]

After:

MemTotal:         372896 kB
00000000-1effffff : System RAM [-0.0M -> 496.0M]
  00000000-00000fff : reserved [-0.0M -> 496.0M]
  00210000-0136ffff : Kernel code [-17.4M -> 478.6M]
  01370000-0183ffff : reserved [-4.8M -> 473.8M]
  01840000-01b7ffff : Kernel data [-3.2M -> 470.6M]
  07140000-0ddfffff : reserved [-108.8M -> 361.8M]
  0de2c000-0deecfff : reserved [-0.8M -> 361.1M]
  0deed000-0df64fff : reserved [-0.5M -> 360.6M]
  0df67000-0df67fff : reserved [-0.0M -> 360.6M]
  0df68000-0df68fff : reserved [-0.0M -> 360.6M]
  0df69000-0df79fff : reserved [-0.1M -> 360.5M]
  0df7a000-1e08dfff : reserved [-257.1M -> 103.4M]
  1e08e000-1e08efff : reserved [-0.0M -> 103.4M]
  1e08f000-1e096fff : reserved [-0.0M -> 103.4M]
  1e097000-1effffff : reserved [-15.4M -> 88.0M]

Here you can clearly see that the third reserved section has gone from 44MB to 108MB, accounting for the ~60MB difference.
(Ah - I had reserved 5MB for kernel logging; without that I get similar numbers to you.)

The obvious next question is what that area is reserved for. The answer is "software IO TLB" - DMA bounce buffers:

[    0.000000] software IO TLB: mapped [mem 0x0000000009600000-0x000000000d600000] (64MB)

This memory is used as a staging post for DMA transfers when the target memory is out of range of the DMA controller. There is no such allocation in the 6.1 kernel, which is puzzling. I'll have to dig deeper.

By the way, the spi1-3cs in dtoverlay=vc4-kms-v3d,spi1-3cs is wrong. It's the name of an overlay, not a parameter, so must be at the start of its own dtoverlay line if want it to have any effect.

pelwell · 2024-02-21T13:59:43Z

The extra 64MB usage is caused by this commit: 370645f

Prior to this, the initialisation of the swiotlb buffers was smart - if all of RAM is addressable by the DMA controller(s) then no bounce buffers are required. This is certainly true on a Z2W with only 512MB, so you save the 64MB carve-out. The above commit, which appeared in Linux 6.5, says that bounce buffers may be required if DMA transfers are not cache-line-aligned, so it is better to have the bounce buffers unconditionally allocated. The DMA_BOUNCE_UNALIGNED_KMALLOC option is turned on by the arm64 Kconfig file. I don't see any way to opt-out without modifying the kernel source - as a config setting without a description it isn't one you can alter from a defconfig file.

Needless to say, the 32-bit (ARCH=arm) Kconfig files don't set DMA_BOUNCE_UNALIGNED_KMALLOC - yet another reason you would be better off with a 32-bit build.

If enabled, DMA_BOUNCE_UNALIGNED_KMALLOC causes the swiotlb buffers (64MB, by default) to be allocated, even on systems where the DMA controller can reach all of RAM. This is a huge amount of RAM to waste on a device with only 512MB to start with, such as the Zero 2 W. See: #5975 Signed-off-by: Phil Elwell <phil@raspberrypi.com>

JamesH65 · 2024-02-21T14:25:33Z

This is an interesting and related read https://lwn.net/Articles/940973/#:~:text=The%20kernel's%20software%20I%2FO,as%20newer%20peripherals%20came%20along.

pelwell · 2024-02-21T14:26:06Z

Given that 6.1 was quite happy without this option being defined, we opted to not enable it on 6.6 - see 3e0065d. This change will be in future releases, but if you wait about 40 minutes (until around 15:10GMT) you should be able to install an early build of this beta kernel by running sudo rpi-update rpi-6.6.y.

For the record, this is the same RAM usage information taken from 6.6 with the patch applied (and without my oversized kernel log buffer):

MemTotal:         476064 kB
00000000-1effffff : System RAM [-0.0M -> 496.0M]
  00000000-00000fff : reserved [-0.0M -> 496.0M]
  00210000-0136ffff : Kernel code [-17.4M -> 478.6M]
  01370000-0183ffff : reserved [-4.8M -> 473.8M]
  01840000-01b7ffff : Kernel data [-3.2M -> 470.6M]
  0d600000-0ddfffff : reserved [-8.0M -> 462.6M]
  0de2c000-0deecfff : reserved [-0.8M -> 461.8M]
  0deed000-0df64fff : reserved [-0.5M -> 461.3M]
  0df67000-0df67fff : reserved [-0.0M -> 461.3M]
  0df68000-0df68fff : reserved [-0.0M -> 461.3M]
  0df69000-0df79fff : reserved [-0.1M -> 461.3M]
  0df7a000-1e08dfff : reserved [-257.1M -> 204.2M]
  1e08e000-1e08efff : reserved [-0.0M -> 204.2M]
  1e08f000-1e096fff : reserved [-0.0M -> 204.1M]
  1e097000-1effffff : reserved [-15.4M -> 188.7M]

popcornmix · 2024-02-21T14:35:24Z

This is an interesting and related read

The patch series described there is in our 6.6 kernel (it wasn't in 6.5 and prior ones).
So it isn't solving this issue.

gugolple · 2024-02-21T14:47:58Z

Given that 6.1 was quite happy without this option being defined, we opted to not enable it on 6.6 - see 3e0065d. This change will be in future releases, but if you wait about 40 minutes (until around 15:10GMT) you should be able to install an early build of this beta kernel by running sudo rpi-update rpi-6.6.y.

Seeing that this is not the only project with this issue, maybe the functionality could be kept with a smaller fix just be setting the kernel boot parameter of "swiotlb=0" as they did in the Yocto project for qemu images.

pelwell · 2024-02-21T14:55:00Z

That's not a bad solution, but this works for us for now.

If enabled, DMA_BOUNCE_UNALIGNED_KMALLOC causes the swiotlb buffers (64MB, by default) to be allocated, even on systems where the DMA controller can reach all of RAM. This is a huge amount of RAM to waste on a device with only 512MB to start with, such as the Zero 2 W. See: #5975 Signed-off-by: Phil Elwell <phil@raspberrypi.com>

See: raspberrypi/linux#5974 kernel: brcmfmac: Fix 802.1x See: raspberrypi/linux#5973 kernel: Add IQaudio CodecZero to hat_map.dts See: raspberrypi/linux#5972 kernel: arm64/Kconfig: Don't set DMA_BOUNCE_UNALIGNED_KMALLOC See: raspberrypi/linux#5975

gugolple · 2024-02-21T20:19:22Z

Can confirm the fix is working.

$ free -h
               total        used        free      shared  buff/cache   available
Mem:           465Mi       161Mi       133Mi       1.4Mi       222Mi       303Mi

$ uname -a
Linux raspberrypi 6.6.17-v8+ #1 SMP PREEMPT Wed Feb 21 16:23:00 UTC 2024 aarch64 GNU/Linux

$ vcgencmd version
Oct 17 2023 15:42:53 
Copyright (c) 2012 Broadcom
version 30f0c5e4d076da3ab4f341d88e7d505760b93ad7 (clean) (release) (start_cd)

Thank you everyone for the support.

pelwell · 2024-02-21T20:42:08Z

Thank you - without your observation all Z2W owners would be missing 12.5% of their RAM for no reason.

If enabled, DMA_BOUNCE_UNALIGNED_KMALLOC causes the swiotlb buffers (64MB, by default) to be allocated, even on systems where the DMA controller can reach all of RAM. This is a huge amount of RAM to waste on a device with only 512MB to start with, such as the Zero 2 W. See: #5975 Signed-off-by: Phil Elwell <phil@raspberrypi.com>

popcornmix transferred this issue from raspberrypi/firmware Feb 20, 2024

pelwell closed this as completed Feb 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RPI Zero 2W incorrect memory size #5975

RPI Zero 2W incorrect memory size #5975

gugolple commented Feb 20, 2024

gugolple commented Feb 20, 2024

popcornmix commented Feb 20, 2024

pelwell commented Feb 20, 2024

gugolple commented Feb 20, 2024

gugolple commented Feb 20, 2024

popcornmix commented Feb 20, 2024

pelwell commented Feb 20, 2024

pelwell commented Feb 20, 2024 •

edited

Loading

gugolple commented Feb 20, 2024

gugolple commented Feb 20, 2024

popcornmix commented Feb 20, 2024

gugolple commented Feb 20, 2024

popcornmix commented Feb 20, 2024

gugolple commented Feb 20, 2024

pelwell commented Feb 21, 2024

pelwell commented Feb 21, 2024 •

edited

Loading

JamesH65 commented Feb 21, 2024

pelwell commented Feb 21, 2024

popcornmix commented Feb 21, 2024

gugolple commented Feb 21, 2024

pelwell commented Feb 21, 2024

gugolple commented Feb 21, 2024

pelwell commented Feb 21, 2024

RPI Zero 2W incorrect memory size #5975

RPI Zero 2W incorrect memory size #5975

Comments

gugolple commented Feb 20, 2024

gugolple commented Feb 20, 2024

popcornmix commented Feb 20, 2024

pelwell commented Feb 20, 2024

gugolple commented Feb 20, 2024

gugolple commented Feb 20, 2024

popcornmix commented Feb 20, 2024

pelwell commented Feb 20, 2024

pelwell commented Feb 20, 2024 • edited Loading

gugolple commented Feb 20, 2024

gugolple commented Feb 20, 2024

popcornmix commented Feb 20, 2024

gugolple commented Feb 20, 2024

popcornmix commented Feb 20, 2024

gugolple commented Feb 20, 2024

pelwell commented Feb 21, 2024

pelwell commented Feb 21, 2024 • edited Loading

JamesH65 commented Feb 21, 2024

pelwell commented Feb 21, 2024

popcornmix commented Feb 21, 2024

gugolple commented Feb 21, 2024

pelwell commented Feb 21, 2024

gugolple commented Feb 21, 2024

pelwell commented Feb 21, 2024

pelwell commented Feb 20, 2024 •

edited

Loading

pelwell commented Feb 21, 2024 •

edited

Loading