F2FS-fs (mmcblk0p3): access invalid blkaddr:259

[f00b4r0]

Writting a special inode to the F2FS filesystem (via e.g. mknod or copying a device node from /dev) will corrupt the filesystem.

If I read the commit message correctly this bug should already be fixed by dda9f4b, however I am running:

$ uname -a
Linux rPi2 4.14.98-v7+ #1200 SMP Tue Feb 12 20:27:48 GMT 2019 armv7l GNU/Linux

And I've been bitten by this issue. I am unable to remove any of the created special files which appear in ls as:

 ls -l chroot/dev/
ls: cannot access 'chroot/dev/null': Bad address
ls: cannot access 'chroot/dev/random': Bad address
total 0
c????????? ? ? ? ?            ? null
c????????? ? ? ? ?            ? random

dmesg output is then filled with:

[321687.954285] F2FS-fs (mmcblk0p3): access invalid blkaddr:264
[321687.954296] ------------[ cut here ]------------
[321687.954316] WARNING: CPU: 3 PID: 31161 at fs/f2fs/checkpoint.c:150 f2fs_is_valid_blkaddr+0x19c/0x1c0
[321687.954322] Modules linked in: ip_set_hash_ip xt_set ip_set nfnetlink xt_CT iptable_raw iptable_nat nf_nat_ipv4 nf_nat xt_limit nf_conntrack_ipv4 nf_defrag_ipv4 xt_tcpudp xt_conntrack nf_conntrack iptable_filter uio_pdrv_genirq uio fixed i2c_dev ip_tables x_tables
[321687.954437] CPU: 3 PID: 31161 Comm: ls Tainted: G        W       4.14.98-v7+ #1200
[321687.954444] Hardware name: BCM2835
[321687.954469] [<8010ff30>] (unwind_backtrace) from [<8010c174>] (show_stack+0x20/0x24)
[321687.954489] [<8010c174>] (show_stack) from [<8078cf04>] (dump_stack+0xd4/0x118)
[321687.954507] [<8078cf04>] (dump_stack) from [<8011dd70>] (__warn+0xf8/0x110)
[321687.954523] [<8011dd70>] (__warn) from [<8011de58>] (warn_slowpath_null+0x30/0x38)
[321687.954541] [<8011de58>] (warn_slowpath_null) from [<8041eaec>] (f2fs_is_valid_blkaddr+0x19c/0x1c0)
[321687.954562] [<8041eaec>] (f2fs_is_valid_blkaddr) from [<8040dc4c>] (f2fs_iget+0x560/0xe98)
[321687.954581] [<8040dc4c>] (f2fs_iget) from [<804111d4>] (f2fs_lookup+0xc0/0x1ac)
[321687.954602] [<804111d4>] (f2fs_lookup) from [<80296e38>] (lookup_slow+0x9c/0x164)
[321687.954623] [<80296e38>] (lookup_slow) from [<80299844>] (walk_component+0x1d8/0x2d4)
[321687.954643] [<80299844>] (walk_component) from [<80299ec0>] (path_lookupat+0x70/0x204)
[321687.954664] [<80299ec0>] (path_lookupat) from [<8029bbec>] (filename_lookup+0x9c/0xf8)
[321687.954684] [<8029bbec>] (filename_lookup) from [<8029bd30>] (user_path_at_empty+0x50/0x58)
[321687.954703] [<8029bd30>] (user_path_at_empty) from [<80290b84>] (vfs_statx+0x68/0xc8)
[321687.954720] [<80290b84>] (vfs_statx) from [<80291578>] (SyS_lstat64+0x38/0x54)
[321687.954740] [<80291578>] (SyS_lstat64) from [<80108000>] (ret_fast_syscall+0x0/0x28)
[321687.954749] ---[ end trace ca85b503f5f1d889 ]---

To reproduce this issue, create a f2fs filesystem, create special inodes on that filesystem, unmount/remount or reboot and observe corruption.

System

• Which model of Raspberry Pi? e.g. Pi3B+, PiZeroW
  Reproduced on Pi1 revA, Pi2 and Pi3B+

• Which OS and version (cat /etc/rpi-issue)?

Raspberry Pi reference 2016-05-27
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 5add7283d4db4eaad785de41a6750e909e5a04cf, stage2

• Which firmware version (vcgencmd version)?

Feb 12 2019 19:43:28
Copyright (c) 2012 Broadcom
version 8eff5e4023657a8b3b59e1f90dc966f62d74908c (clean) (release) (start_cd)

• Which kernel version (uname -a)?
  Linux rPi2 4.14.98-v7+ #1200 SMP Tue Feb 12 20:27:48 GMT 2019 armv7l GNU/Linux

Hope this helps

[6by9]

The patch you reference is in 4.19, whilst you're running 4.14. If those upstream have not tagged the fix as required for the stable/LTS branches, then it won't be backported.

These are unmodified upstream files, therefore issues really ought to be flagged to the normal kernel mailing lists.

[Phiber2000]

Same problem here (with different blkaddr) using the kernel of 1.20190215.
Upgrading to the current 4.19 kernel build or downgrading to 1.20181112 fixes the issue.
I always get the error when accessing the /var/spool/postfix/dev/ directory which contains two character devices.

[Phiber2000]

I had a look at the current stable 4.14.106 at kernel.org. The issue is still present here.
@chaseyu Please could you have a look at it? I think we need a backport of dda9f4b9@kernel.org.

[pelwell]

We will consider a back-port of this commit if someone can provide confirmation that it fixes the problem.

[Phiber2000]

Kernel build and confirmed fully working again! The character devices are fully working and no problems are shown in syslog anymore.

git diff from raspberrypi-kernel_1.20190215-1

diff --git a/fs/f2fs/inode.c b/fs/f2fs/inode.c
index 50818b519..4898570ac 100644
--- a/fs/f2fs/inode.c
+++ b/fs/f2fs/inode.c
@@ -310,13 +310,15 @@ static int do_read_inode(struct inode *inode)
        /* get rdev by using inline_info */
        __get_inode_rdev(inode, ri);
 
-       err = __written_first_block(sbi, ri);
-       if (err < 0) {
-               f2fs_put_page(node_page, 1);
-               return err;
+       if (S_ISREG(inode->i_mode)) {
+               err = __written_first_block(sbi, ri);
+               if (err < 0) {
+                       f2fs_put_page(node_page, 1);
+                       return err;
+               }
+               if(!err)
+                       set_inode_flag(inode, FI_FIRST_BLOCK_WRITTEN);
        }
-       if (!err)
-               set_inode_flag(inode, FI_FIRST_BLOCK_WRITTEN);
 
        if (!need_inode_block_update(sbi, inode->i_ino))
                fi->last_disk_size = inode->i_size;

Test results

user@pi:~$ uname -a
Linux pi 4.14.98-v7+ #1 SMP Mon Mar 18 21:14:41 CET 2019 armv7l GNU/Linux
user@pi:~$ ls -l /var/spool/postfix/dev/
total 0
srw-rw-rw- 1 root root    0 Mar 18 22:07 log
crw-rw-rw- 1 root root 1, 8 Mar 18 22:07 random
crw-rw-rw- 1 root root 1, 9 Mar 18 22:07 urandom
user@pi:~$ head -c10 /var/spool/postfix/dev/random
vS▒T!攭>

Artifacts

kernels_mr.zip (build and tested from merge request)

[Phiber2000]

Thanks to @f00b4r0 for addressing the reason! I hope this will be part of the next kernel release.

[pelwell]

#2896 is a PR of the back-port to 4.14, which I am prepared to merge, and the 4.14 kernel is a Long Term Stable version, but as far as we are concerned it is End Of Life. Are you sure that upgrading to 4.19 isn't a better option for you?

[Phiber2000]

First: Thank's a lot for your effort!

To your suggestion:
Many users, like me, use the repository at archive.raspberrypi.org. That's where we got the semi-broken F2FS kernel from.
If this official repository gets the 4.19, apt will do its job. I don't want to break the package based kernel upgrade...
I hope you can understand this.

[pelwell]

sudo rpi-update will get you an "official" 4.19 build, and I've been told that future Raspbian releases are going to be based on 4.19, so unless you have a specific reason to want a 4.14 kernel you would be better off switching. However, as this patch is clearly a bug fix, I've merged in case it benefits users who build from our source trees and are reluctant to upgrade.

[f00b4r0]

(Chiming in after being AFK). Thanks @Phiber2000 for testing and confirming the patch works.

@pelwell quick - hopefully useful - remarks:

rpi-update is not a mandatory tool (it's classified "Priority: extra"), and it's also fairly orthogonal to the "usual way" of handling debian-based distributions.

As far as I'm concerned it's not installed on any of my raspbian setups, and considering 1) it's old (2014), 2) it's pulling from some "random" git repo without any kind of cryptographic signature check (AFAICT), I consider it a security risk (and I suspect I'm not alone). The tool appears to be operating completely oblivious to the way apt/dpkg manage files and is likely to break kernel package management eventually.

My 2c: rpi-update should be obsoleted and kernel/firmware should be provided via standard packages. I don't know if there are/will be distros based on raspbian but that would also likely make that easier...

HTH

[pelwell]

Raspberry Pi will not be releasing any more 4.14 kernel builds - it's all 4.19 from here on.

[JamesH65]

rpi-update is a Pi supplied tool for grabbing testing version of kernel and firmware. Although we have been using it since the first Pi, we no longer recommend it for general use (well, never have done really), only for those people wanting to get bleeding edge test firmware and kernels for testing reasons. I'm not sure the usual package management system would be suitable for that.

[f00b4r0]

@JamesH65 thanks for the clarification. Should there ever be a need to do what rpi-update does (testing bleeding edge firmware and kernels) the "debian way", I believe the correct approach would be to use the "experimental" branch, which is designed for exactly that purpose.

@pelwell that's ok, but these kernels should be provided via apt, imho. Please don't discount the fact that people who run the (last) "stable" release of raspbian are usually doing so for security and stability reasons.
Speaking of which I've noticed that raspbian doesn't seem to provide kernel security updates for past releases (e.g. jessie). While I understand that this is probably a consequence of lack of manpower, I would recommend that this fact be more clearly advertised (I couldn't find any mention of that anywhere on rasbpian.org or raspberrypi.org). There are a lot of Pis deployed in the wild, many have an online presence and it'd be a shame to see them compromised because they're running on flawed kernels... Again, my humble 2c, and sorry for hijacking this ticket with my random thoughts :)

[JamesH65]

I suspect the overhead of maintaining an experimental branch, vs the extremely simple github mechanism we use at the moment would means this is a non-starter. The current system also allows very easy movement between kernel/firmware combinations via git hashes. Does the experimental branch approach provide that?

With regard to older kernel security, this is mostly a manpower issue. We always recommend usage of our latest releases to ensure the best security.

[f00b4r0]

The current system also allows very easy movement between kernel/firmware combinations via git hashes. Does the experimental branch approach provide that?

Probably not as easily.

With regard to older kernel security, this is mostly a manpower issue. We always recommend usage of our latest releases to ensure the best security.

OK, that makes sense. Still, maybe I missed something but I couldn't find direct, obvious recommendation to that end on the websites. I think there's a gap between "we recommend using our latest releases to ensure the best security" and "we don't provide (kernel) security updates on our previous releases": imho the former doesn't imply the latter.

Best

[Phiber2000]

I think, we can close this issue here. I left a hint at https://bugzilla.kernel.org/show_bug.cgi?id=202495 and mentioned one of the devs here last week to get that fixed there too. Let's see, if they move as quick as the devs and maintainers here.

[f00b4r0]

Hi, is there an ETA on getting this fix into the Stretch repository? Thanks