encrypted swap causes system to crash/reboot

[kapitainsky]

Describe the bug
swap partition encrypted with cryptsetup causes system to reboot as soon as swap is used

To reproduce

In this example there is external disk attached with 2GB swap partition (/dev/sda2)

disable existing swap:
sudo swapoff -a

install cryptsetup:
sudo apt-get install cryptsetup

edit /etc/crypttab and add:
cryptswap /dev/sda2 /dev/urandom swap,cipher=aes-xts-plain64,size=256,hash=sha1,noauto

start encrypted partition:
sudo cryptdisks_start cryptswap

check that above worked:

pi@raspberrypi:~$ ll /dev/mapper
total 0
crw------- 1 root root 10, 236 Oct 16 12:28 control
lrwxrwxrwx 1 root root       7 Oct 16 12:48 cryptswap -> ../dm-0

pi@raspberrypi:~$ sudo cryptsetup status /dev/mapper/cryptswap
/dev/mapper/cryptswap is active.
  type:    PLAIN
  cipher:  aes-xts-plain64
  keysize: 256 bits
  key location: dm-crypt
  device:  /dev/sda2
  sector size:  512
  offset:  0 sectors
  size:    3911647 sectors
  mode:    read/write

enable encrypted swap:
sudo swapon /dev/mapper/cryptswap

check that above worked:

pi@raspberrypi:~$ sudo swapon -s
Filename                                Type            Size    Used    Priority
/dev/dm-0                               partition       1955816 0       -2

make use of it, I force it with:

stress -m 2 --vm-bytes 500M

but initially I noticed it during normal usage e.g. when running complex software build.

I observe swap usage using htop

Expected behaviour
system should use encrypted swap.

Actual behaviour
system reboots as soon as swap is being used

System

• Which model of Raspberry Pi? tested with:
  2B, 3 and 3B+

• Which OS and version (cat /etc/rpi-issue)?
  Raspberry Pi reference 2019-09-26
  Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 80d486687ea77d31fc3fc13cf3a2f8b464e129be, stage2

• Which firmware version (vcgencmd version)?
  Sep 24 2019 17:37:47
  Copyright (c) 2012 Broadcom
  version 6820edeee4ef3891b95fc01cf02a7abd7ca52f17 (clean) (release) (start_cd)

• Which kernel version (uname -a)?
  Linux raspberrypi 4.19.75-v7+ solved issue of mirroring screen after rotation. #1270 SMP Tue Sep 24 18:45:11 BST 2019 armv7l GNU/Linux

Logs
There is nothing in logs related to reboot, no OOM... nothing
The last entries in kern.log

Oct 16 12:28:18 raspberrypi kernel: [  106.718202] device-mapper: ioctl: 4.39.0-ioctl (2018-04-03) initialised: dm-devel@redhat.com
Oct 16 12:28:18 raspberrypi kernel: [  107.208600] cryptd: max_cpu_qlen set to 1000
Oct 16 12:28:30 raspberrypi kernel: [  119.598168] Adding 1955816k swap on /dev/mapper/cryptswap.  Priority:-3 extents:1 across:1955816k FS

followed by fresh boot sequence.

Additional context
When swap partition is not using encryption everything works perfectly. I have tried external HDD, memory sticks etc. Using partition or full disk. Results are always the same - as soon as I try encrypt swap system reboots.

To rule out any misconfiguration I have run it with the latest fresh install of raspbian. All updates applied.

I have raised this issue on raspberry pi forum (Encrypted swap crashes all system) but so far no solution/cause have been found.

[JamesH65]

When trying this with an encrypted swap FILE, I got the following kernel dump via a serial console.

[  143.156303] swapper/0: page allocation failure: order:0, mode:0x480020(GFP_ATOMIC), nodemask=(null)
[  143.156310] swapper/0 cpuset=/ mems_allowed=0
[  143.156324] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G         C        4.19.75-v7l+ #1270
[  143.156327] Hardware name: BCM2835
[  143.156349] [<c0212d10>] (unwind_backtrace) from [<c020d530>] (show_stack+0x20/0x24)
[  143.156359] [<c020d530>] (show_stack) from [<c097fb20>] (dump_stack+0xd4/0x118)
[  143.156367] [<c097fb20>] (dump_stack) from [<c0344638>] (warn_alloc+0xcc/0x170)
[  143.156376] [<c0344638>] (warn_alloc) from [<c03457dc>] (__alloc_pages_nodemask+0x104c/0x1180)
[  143.156382] [<c03457dc>] (__alloc_pages_nodemask) from [<c0345b00>] (page_frag_alloc+0x14c/0x160)
[  143.156391] [<c0345b00>] (page_frag_alloc) from [<c0854d44>] (__netdev_alloc_skb+0xb4/0x158)
[  143.156403] [<c0854d44>] (__netdev_alloc_skb) from [<c073d1a4>] (bcmgenet_rx_refill+0x44/0x264)
[  143.156412] [<c073d1a4>] (bcmgenet_rx_refill) from [<c073d900>] (bcmgenet_rx_poll+0x254/0xa3c)
[  143.156419] [<c073d900>] (bcmgenet_rx_poll) from [<c086f890>] (net_rx_action+0x310/0x494)
[  143.156427] [<c086f890>] (net_rx_action) from [<c0202410>] (__do_softirq+0x190/0x3f0)
[  143.156435] [<c0202410>] (__do_softirq) from [<c0228308>] (irq_exit+0xfc/0x120)
[  143.156445] [<c0228308>] (irq_exit) from [<c0280e08>] (__handle_domain_irq+0x70/0xc4)
[  143.156453] [<c0280e08>] (__handle_domain_irq) from [<c0202244>] (gic_handle_irq+0x4c/0x88)
[  143.156459] [<c0202244>] (gic_handle_irq) from [<c02019bc>] (__irq_svc+0x5c/0x7c)
[  143.156463] Exception stack(0xc1001ed8 to 0xc1001f20)
[  143.156467] 1ec0:                                                       c0209a84 00000000
[  143.156472] 1ee0: 40000193 40000193 c1004dbc c1000000 c1004e04 00000001 c109641a c10a56c0
[  143.156477] 1f00: c0e67a38 c1001f34 c10051c4 c1001f28 00000000 c0209a88 40000113 ffffffff
[  143.156484] [<c02019bc>] (__irq_svc) from [<c0209a88>] (arch_cpu_idle+0x34/0x4c)
[  143.156491] [<c0209a88>] (arch_cpu_idle) from [<c099cd2c>] (default_idle_call+0x34/0x48)
[  143.156500] [<c099cd2c>] (default_idle_call) from [<c02542b4>] (do_idle+0xec/0x17c)
[  143.156509] [<c02542b4>] (do_idle) from [<c0254604>] (cpu_startup_entry+0x28/0x2c)
[  143.156516] [<c0254604>] (cpu_startup_entry) from [<c09963c0>] (rest_init+0xbc/0xc0)
[  143.156526] [<c09963c0>] (rest_init) from [<c0e00fe4>] (start_kernel+0x4b8/0x4e8)
[  143.156531] Mem-Info:
[  143.156539] active_anon:883724 inactive_anon:24732 isolated_anon:11
                active_file:17532 inactive_file:20998 isolated_file:37
                unevictable:4 dirty:0 writeback:73 unstable:0
                slab_reclaimable:4261 slab_unreclaimable:7671
                mapped:18022 shmem:12516 pagetables:2532 bounce:68
                free:7131 free_pcp:63 free_cma:5866
[  143.156547] Node 0 active_anon:3534896kB inactive_anon:98928kB active_file:70128kB inactive_file:83992kB unevictable:16kB isolated(anon):44kB isolated(file):148kB mapped:72088kB dirty:0kB writeback:292kB shmem:50064kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[  143.156555] DMA free:28472kB min:16384kB low:20480kB high:24576kB active_anon:570980kB inactive_anon:0kB active_file:14824kB inactive_file:7100kB unevictable:0kB writepending:0kB present:786432kB managed:735740kB mlocked:0kB kernel_stack:1736kB pagetables:692kB bounce:272kB free_pcp:0kB local_pcp:0kB free_cma:23464kB
[  143.156558] lowmem_reserve[]: 0 0 3136 3136
[  143.156579] HighMem free:52kB min:512kB low:18388kB high:36264kB active_anon:2963500kB inactive_anon:98164kB active_file:56380kB inactive_file:77232kB unevictable:16kB writepending:292kB present:3211264kB managed:3211264kB mlocked:16kB kernel_stack:0kB pagetables:9436kB bounce:0kB free_pcp:252kB local_pcp:252kB free_cma:0kB
[  143.156581] lowmem_reserve[]: 0 0 0 0
[  143.156597] DMA: 188*4kB (UMEC) 46*8kB (UMEC) 32*16kB (UMEC) 18*32kB (UMEC) 10*64kB (UEC) 4*128kB (U) 0*256kB 2*512kB (EC) 0*1024kB 2*2048kB (EC) 5*4096kB (C) = 28960kB
[  143.156658] HighMem: 5*4kB (UM) 16*8kB (UM) 7*16kB (U) 1*32kB (M) 3*64kB (M) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 612kB
[  143.156709] 51199 total pagecache pages
[  143.156713] 115 pages in swap cache
[  143.156716] Swap cache stats: add 169, delete 54, find 0/0
[  143.156719] Free swap  = 625400kB
[  143.156722] Total swap = 626680kB
[  143.156725] 999424 pages RAM
[  143.156728] 802816 pages HighMem/MovableOnly
[  143.156731] 12673 pages reserved
[  143.156734] 65536 pages cma reserved
[  143.477671] swapper/0: page allocation failure: order:0, mode:0x480020(GFP_ATOMIC), nodemask=(null)
[  143.477677] swapper/0 cpuset=/ mems_allowed=0
[  143.477692] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G         C        4.19.75-v7l+ #1270
[  143.477695] Hardware name: BCM2835
[  143.477716] [<c0212d10>] (unwind_backtrace) from [<c020d530>] (show_stack+0x20/0x24)
[  143.477727] [<c020d530>] (show_stack) from [<c097fb20>] (dump_stack+0xd4/0x118)
[  143.477736] [<c097fb20>] (dump_stack) from [<c0344638>] (warn_alloc+0xcc/0x170)
[  143.477745] [<c0344638>] (warn_alloc) from [<c03457dc>] (__alloc_pages_nodemask+0x104c/0x1180)
[  143.477751] [<c03457dc>] (__alloc_pages_nodemask) from [<c0345b00>] (page_frag_alloc+0x14c/0x160)
[  143.477760] [<c0345b00>] (page_frag_alloc) from [<c0854d44>] (__netdev_alloc_skb+0xb4/0x158)
[  143.477771] [<c0854d44>] (__netdev_alloc_skb) from [<c073d1a4>] (bcmgenet_rx_refill+0x44/0x264)
[  143.477780] [<c073d1a4>] (bcmgenet_rx_refill) from [<c073d900>] (bcmgenet_rx_poll+0x254/0xa3c)
[  143.477787] [<c073d900>] (bcmgenet_rx_poll) from [<c086f890>] (net_rx_action+0x310/0x494)
[  143.477795] [<c086f890>] (net_rx_action) from [<c0202410>] (__do_softirq+0x190/0x3f0)
[  143.477804] [<c0202410>] (__do_softirq) from [<c0228308>] (irq_exit+0xfc/0x120)
[  143.477814] [<c0228308>] (irq_exit) from [<c0280e08>] (__handle_domain_irq+0x70/0xc4)
[  143.477822] [<c0280e08>] (__handle_domain_irq) from [<c0202244>] (gic_handle_irq+0x4c/0x88)
[  143.477828] [<c0202244>] (gic_handle_irq) from [<c02019bc>] (__irq_svc+0x5c/0x7c)
[  143.477832] Exception stack(0xc1001ed8 to 0xc1001f20)
[  143.477836] 1ec0:                                                       c0209a84 00000000
[  143.477841] 1ee0: 40000193 40000193 c1004dbc c1000000 c1004e04 00000001 c109641a c10a56c0
[  143.477846] 1f00: c0e67a38 c1001f34 c10051c4 c1001f28 00000000 c0209a88 40000113 ffffffff
[  143.477853] [<c02019bc>] (__irq_svc) from [<c0209a88>] (arch_cpu_idle+0x34/0x4c)
[  143.477860] [<c0209a88>] (arch_cpu_idle) from [<c099cd2c>] (default_idle_call+0x34/0x48)
[  143.477869] [<c099cd2c>] (default_idle_call) from [<c02542b4>] (do_idle+0xec/0x17c)
[  143.477877] [<c02542b4>] (do_idle) from [<c0254604>] (cpu_startup_entry+0x28/0x2c)
[  143.477884] [<c0254604>] (cpu_startup_entry) from [<c09963c0>] (rest_init+0xbc/0xc0)
[  143.477894] [<c09963c0>] (rest_init) from [<c0e00fe4>] (start_kernel+0x4b8/0x4e8)

[kapitainsky]

from my Google encrypted swap FILE does not work in general. encrypted swap PARTITION should work. But never found what is the problem with files hence my focus was on encrypting partition only. Swap FILE without encryption works perfectly.

[JamesH65]

So I am wasting my time with testing this on a swap file - i need to use a swap partition? Getting to the point of not worth the time for a minimal use case.

[KrisztianKende]

So I am wasting my time with testing this on a swap file - i need to use a swap partition? Getting to the point of not worth the time for a minimal use case.

Sorry, I can confirm that the encrypted SWAP file is not working. If we want to get ahead, we also need to test with a partition. I tested it on a Pi4 with 64-bit kernel, but works fine.

[JamesH65]

Taken a brief look at this, just enough to confirm that I won't be able to figure out what is going on in a sane timescale - very much out of my area of expertise - needs a linux memory management/encryption expert.

The fact it works on 64bit might indicate a problem with the encryption path in 32bit, but I have no idea how to track that down. Worth noting there is no HW encryption on the Pi SoC, so this is always going to be slow anyway as its done in SW - there could even be timeouts that might cause problems.

[kapitainsky]

Thank you for looking into it.

Hopefully 64bits raspbian will be ready for general release soon and if it fixes this and other issues better to focus on it.

cryptsetup dev have similar suspicions https://gitlab.com/cryptsetup/cryptsetup/issues/495

[hypnotoad]

I have the same problem with a 64 bit kernel and a non-encrypted swap file:
Linux tensing 4.19.75-v8+ #1270 SMP PREEMPT Tue Sep 24 18:59:17 BST 2019 aarch64 GNU/Linux

However, I have an encyrpted partition. Could it maybe be that the network driver gets lots of input and cannot get a buffer since the write back buffer fills up too much as encryption and storage to disk are too slow?

[Sa Nov 23 12:52:30 2019] ksoftirqd/0: page allocation failure: order:0, mode:0x480020(GFP_ATOMIC), nodemask=(null)
[Sa Nov 23 12:52:30 2019] ksoftirqd/0 cpuset=/ mems_allowed=0
[Sa Nov 23 12:52:30 2019] CPU: 0 PID: 9 Comm: ksoftirqd/0 Tainted: G         C        4.19.75-v8+ #1270
[Sa Nov 23 12:52:30 2019] Hardware name: Raspberry Pi 4 Model B Rev 1.1 (DT)
[Sa Nov 23 12:52:30 2019] Call trace:
[Sa Nov 23 12:52:30 2019]  dump_backtrace+0x0/0x178
[Sa Nov 23 12:52:30 2019]  show_stack+0x24/0x30
[Sa Nov 23 12:52:30 2019]  dump_stack+0x9c/0xd4
[Sa Nov 23 12:52:30 2019]  warn_alloc+0xec/0x158
[Sa Nov 23 12:52:30 2019]  __alloc_pages_nodemask+0xd80/0xe38
[Sa Nov 23 12:52:30 2019]  page_frag_alloc+0x12c/0x158
[Sa Nov 23 12:52:30 2019]  __netdev_alloc_skb+0xd8/0x160
[Sa Nov 23 12:52:30 2019]  bcmgenet_rx_refill+0x44/0x248
[Sa Nov 23 12:52:30 2019]  bcmgenet_rx_poll+0x274/0x8a0
[Sa Nov 23 12:52:30 2019]  net_rx_action+0x2e4/0x410
[Sa Nov 23 12:52:30 2019]  __do_softirq+0x17c/0x3cc
[Sa Nov 23 12:52:30 2019]  run_ksoftirqd+0x4c/0x60
[Sa Nov 23 12:52:30 2019]  smpboot_thread_fn+0x1f8/0x298
[Sa Nov 23 12:52:30 2019]  kthread+0x104/0x130
[Sa Nov 23 12:52:30 2019]  ret_from_fork+0x10/0x18

[hypnotoad]

So after executing the following command, the problem does not happen any more for me (and it happened more than once per day before):

sysctl -w vm.min_free_kbytes=65536

This command does not make the setting permanent over reboots - there are various config files to make it permament.

[kapitainsky]

So after executing the following command, the problem does not happen any more for me (and it happened more than once per day before):

sysctl -w vm.min_free_kbytes=65536

This command does not make the setting permanent over reboots - there are various config files to make it permament.

problem with encrypted swap? on 32 aor 64 bits raspbain?

[hypnotoad]

I had NO encrypted swap, but an encrypted partition (swap on unencrypted usb stick).

I would bet that the problem is not the swap, but the encryption in general when he receives too much data over this kernel driver.

Before, I had problems on an Rpi 4, Raspian Buster, both 32 as well as 64 bit kernels.

The fix helped for 4.19.75-v8+ #1270 SMP PREEMPT Tue Sep 24 18:59:17 BST 2019 aarch64 GNU/Linux

[pelwell]

Is it not complaining about a spurious '#' after the 65536?

[hypnotoad]

I execute the following command and receive only 1 line of output:

root@pi$ sysctl -w vm.min_free_kbytes=65536
vm.min_free_kbytes = 65536

Still no problem after 46 days of uptime (I had problems after 1-10 days before).