syzbot: enable UBSAN for linux

[dvyukov]

Tracking issue for enabling KUBSAN for linux on syzbot.
Moved from the discussion on a patch:
https://groups.google.com/g/syzkaller/c/Ex_s_aokx0A/m/AfzioXHEAwAJ

Things we know we need:

1. panic_on_warn support would be nice to land, but not critical.

2. If we want to selectively enable checks, then we need to wait for CONFIG_UBSAN_MISC support. @kees proposed the following config:

CONFIG_UBSAN=y
CONFIG_UBSAN_BOUNDS=y
# CONFIG_UBSAN_MISC is not set

But I've just tested latest linux-next with already supported:

CONFIG_UBSAN=y
CONFIG_UBSAN_SANITIZE_ALL=y
CONFIG_UBSAN_NO_ALIGNMENT=y

and I don't see any massive reports. So perhaps we enable everything (except for alignment?) right away? No need to wait for the patches then.
Update: with a bit more testing I found 3 invalid shift exponents in ext4, nfs and f2fs mount syscall.

3. Figure out what we do with bug titles.
   Current we just say "UBSAN: undefined-behaviour":
   https://github.com/google/syzkaller/blob/master/pkg/report/testdata/linux/report/453
   UBSAN does not give a sane bug type identity, for bug type it produces lengthy prose with lots of moving parts:

index 6380 is out of range for type 's8 [257]'
index 8 is out of range for type 'char [8]'
signed integer overflow: -1531247276 cannot be represented in type 'int'

which is not suitable as part of bug identity.
We have 2 options: (1) leave it as-is (just say undefined-behaviour) or (2) rework UBSAN bug reporting to provide sane bug type.

4. UBSAN vs KASAN precedence.
   It turns out UBSAN steals bugs from KASAN. E.g. this one:
   https://github.com/google/syzkaller/blob/master/pkg/report/testdata/linux/report/454
   is already detected and reported by KASAN. This is quite unfortunate b/c we may start reporting lots of dups if we enable UBSAN. I don't have good ideas so far.

[dvyukov]

FTR, UBSAN improvements from @kees were merged mainline, in particular:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ef065653e526a020d2a71549f413f14a830db799

[kees]

I think we're ready to move forward, which some discussion on point 4:

1. panic_on_warn was added
2. bounds checking has been split out
3. bug titles are fixed, as you noted above
4. I'm not sure how to deal with collisions between UBSAN and KASAN, but it seems like UBSAN might trip earlier (which might make things easier to debug), but KASAN includes function names (which can make reports faster to analyze). The referenced example was "index 6380 is out of range for type 's8 [257]'". Was that more or less useful than what KASAN reported? (I can't find what KASAN said about this flaw...)

Regardless, I think the follow Kconfig would be the best first step:

CONFIG_UBSAN=y
# CONFIG_UBSAN_TRAP is not set
CONFIG_UBSAN_BOUNDS=y
# CONFIG_UBSAN_MISC is not set
CONFIG_UBSAN_SANITIZE_ALL=y
# CONFIG_UBSAN_ALIGNMENT is not set

This will get us bounds checking (which from my testing has a low false-positive rate). I'd like to split arithmetic overflow instrumentation out of CONFIG_UBSAN_MISC in the future, but it still has several other problems to solve first:
KSPP/linux#26
KSPP/linux#27

[dvyukov]

I'm not sure how to deal with collisions between UBSAN and KASAN, but it seems like UBSAN might trip earlier (which might make things easier to debug), but KASAN includes function names (which can make reports faster to analyze). The referenced example was "index 6380 is out of range for type 's8 [257]'". Was that more or less useful than what KASAN reported? (I can't find what KASAN said about this flaw...)

I think they are mostly equivalent. I was more concerned about getting KASAN reports for some bugs, and now start getting UBSAN reports for the same bugs, which will be reported as new bugs. Let's see if it will be a problem or not. Maybe it won't be a problem because we don't seem to detect massive amounts of these UBSAN reports.

Config and report parsing changes are now in (#2120). Let's see what happens :)

[dvyukov]

We now have 2 UBSAN bugs, everything seems to be working as intended.
@kees thanks for working on the kernel part.
If/when we have any new checks to enable, that can be done as in #2120.

[dvyukov]

One issue that come up:
syzbot tried to bisect this UBSAN bug:
https://syzkaller.appspot.com/bug?extid=9b64b619f10f19d19a7c
https://syzkaller.appspot.com/x/bisect.txt?x=1697348d900000
and then diverged into another bug as in the past we had more checks enabled and there were no way to selectively enable them, so it started hitting all possible minor ones.
Additionally UBSAN was build-broken for 4 releases v4.11-v4.15.
Potentially we need to restrict bisection of UBSAN bugs down to v5.3 only.

[dvyukov]

Another bisection broken broken by UBSAN:
https://syzkaller.appspot.com/text?tag=Log&x=1474aaad900000

testing release v5.7
testing commit 3d77e6a8804abcc0504c904bd6e5cdf3a5cf8162 with gcc (GCC) 8.1.0
kernel signature: b9f790ec4d2c637835f5e7459cddc3e9bde402f4e9d8dea72d0ed641d75d9b9e
all runs: crashed: WARNING in sta_info_insert_rcu
testing release v5.6
testing commit 7111951b8d4973bda27ff663f2cf18b663d15b48 with gcc (GCC) 8.1.0
kernel signature: 07eaee35dc2e497dac2ca4af30768de72b406f3e4842fa07285b89a1105b6cf4
all runs: crashed: WARNING in sta_info_insert_rcu
testing release v5.5
testing commit d5226fa6dbae0569ee43ecfc08bdcd6770fc4755 with gcc (GCC) 8.1.0
kernel signature: 452351e257e8a0cc0748f5ef79e2673d8a416414dbfa67c39ded8d46ae415d30
all runs: crashed: WARNING in sta_info_insert_rcu
testing release v5.4
testing commit 219d54332a09e8d8741c1e1982f5eae56099de85 with gcc (GCC) 8.1.0
kernel signature: bb778761c5ef6430c6eaa5685fbf33401a496bbffa08e69f398ce9d2484e1f47
all runs: crashed: WARNING in sta_info_insert_rcu
testing release v5.3
testing commit 4d856f72c10ecb060868ed10ff1b1453943fc6c8 with gcc (GCC) 8.1.0
kernel signature: 9a1d4f2fc01569bfccf5662e9c4e79a2e71455f21412380dc2a8ee9b972b40f4
all runs: crashed: WARNING in sta_info_insert_rcu
testing release v5.2
testing commit 0ecfebd2b52404ae0c54a878c872bb93363ada36 with gcc (GCC) 8.1.0
kernel signature: b5ad3f2922691f5a58fd77477a89e62da8a4ba06098e187e9bedb79a92f7e30f
all runs: basic kernel testing failed: UBSAN: undefined-behaviour in vhost_vq_reset
testing release v5.1
testing commit e93c9c99a629c61837d5a7fc2120cd2b6c70dbdd with gcc (GCC) 8.1.0
kernel signature: cba8ed3a9401ac90c4563c482d0e502f97be441f35e6489e7110a0f48ff23245
run #0: basic kernel testing failed: UBSAN: undefined-behaviour in vhost_vq_reset
run #1: basic kernel testing failed: UBSAN: undefined-behaviour in vhost_vq_reset
run #2: basic kernel testing failed: UBSAN: undefined-behaviour in vhost_vq_reset
run #3: basic kernel testing failed: UBSAN: undefined-behaviour in vhost_vq_reset
run #4: basic kernel testing failed: UBSAN: undefined-behaviour in vhost_vq_reset
run #5: basic kernel testing failed: UBSAN: undefined-behaviour in vhost_vq_reset
run #6: basic kernel testing failed: UBSAN: undefined-behaviour in vhost_vq_reset
run #7: basic kernel testing failed: UBSAN: undefined-behaviour in vhost_vq_reset
run #8: basic kernel testing failed: UBSAN: undefined-behaviour in vhost_vq_reset
run #9: crashed: WARNING in sta_info_insert_rcu
testing release v5.0
testing commit 1c163f4c7b3f621efff9b28a47abb36f7378d783 with gcc (GCC) 8.1.0
kernel signature: b4409eaa12702e36eb6220cc3a933a7f7f475461d2a10b3cf652298dc72c3894
all runs: basic kernel testing failed: UBSAN: undefined-behaviour in vhost_vq_reset
testing release v4.20
testing commit 8fe28cb58bcb235034b64cbbb7550a8a43fd88be with gcc (GCC) 8.1.0
kernel signature: 3de6d1b4a72bfacbea8facb65cb2f1f272be8a0597ccda89d43a25464ceb1823
all runs: basic kernel testing failed: UBSAN: undefined-behaviour in vhost_vq_reset
testing release v4.19
testing commit 84df9525b0c27f3ebc2ebb1864fa62a97fdedb7d with gcc (GCC) 8.1.0
kernel signature: 6a7f165c5ac307deb9a5bed5a8ace42b8d6207b2c190f708177f665accbfc7aa
all runs: basic kernel testing failed: UBSAN: undefined-behaviour in vhost_vq_reset
testing release v4.18
testing commit 94710cac0ef4ee177a63b5227664b38c95bbf703 with gcc (GCC) 8.1.0
kernel signature: 22b58a42a53ee7260bd4e8d38f3ef9d1a70177c9afa8c7e2e19bfc331dcd6e96
all runs: basic kernel testing failed: UBSAN: undefined-behaviour in vhost_vq_reset
testing release v4.17
testing commit 29dcea88779c856c7dc92040a0c01233263101d4 with gcc (GCC) 8.1.0
kernel signature: 7938144cb6355a37a96e17e5b4d5dd15197035756fa33386658d3f8738cd71ec
all runs: basic kernel testing failed: timed out
testing release v4.16
testing commit 0adb32858b0bddf4ada5f364a84ed60b196dbcda with gcc (GCC) 8.1.0
kernel signature: 84a4cc7363f83a715f365cbd7445e1f7c4e98fc856d3054f2859c2b45b400bf1
all runs: basic kernel testing failed: UBSAN: undefined-behaviour in vhost_vq_reset
testing release v4.15
testing commit d8a5b80568a9cb66810e75b182018e9edb68e8ff with gcc (GCC) 8.1.0
./arch/x86/include/asm/bootparam_utils.h:37: undefined reference to `__ubsan_handle_type_mismatch_v1'
./arch/x86/include/asm/bootparam_utils.h:39: undefined reference to `__ubsan_handle_type_mismatch_v1'
./arch/x86/include/asm/bootparam_utils.h:42: undefined reference to `__ubsan_handle_type_mismatch_v1'
./arch/x86/include/asm/bootparam_utils.h:45: undefined reference to `__ubsan_handle_type_mismatch_v1'
./arch/x86/include/asm/bootparam_utils.h:48: undefined reference to `__ubsan_handle_type_mismatch_v1'
testing release v4.14
testing commit bebc6082da0a9f5d47a1ea2edc099bf671058bd4 with gcc (GCC) 8.1.0
./arch/x86/include/asm/bootparam_utils.h:37: undefined reference to `__ubsan_handle_type_mismatch_v1'
./arch/x86/include/asm/bootparam_utils.h:39: undefined reference to `__ubsan_handle_type_mismatch_v1'
./arch/x86/include/asm/bootparam_utils.h:42: undefined reference to `__ubsan_handle_type_mismatch_v1'
./arch/x86/include/asm/bootparam_utils.h:45: undefined reference to `__ubsan_handle_type_mismatch_v1'
./arch/x86/include/asm/bootparam_utils.h:48: undefined reference to `__ubsan_handle_type_mismatch_v1'
testing release v4.13
testing commit 569dbb88e80deb68974ef6fdd6a13edb9d686261 with gcc (GCC) 8.1.0
./arch/x86/include/asm/bootparam_utils.h:36: undefined reference to `__ubsan_handle_type_mismatch_v1'
./arch/x86/include/asm/bootparam_utils.h:38: undefined reference to `__ubsan_handle_type_mismatch_v1'
./arch/x86/include/asm/bootparam_utils.h:41: undefined reference to `__ubsan_handle_type_mismatch_v1'
./arch/x86/include/asm/bootparam_utils.h:44: undefined reference to `__ubsan_handle_type_mismatch_v1'
./arch/x86/include/asm/bootparam_utils.h:47: undefined reference to `__ubsan_handle_type_mismatch_v1'
testing release v4.12
testing commit 6f7da290413ba713f0cdd9ff1a2a9bb129ef4f6c with gcc (GCC) 8.1.0
arch/x86/entry/common.c:173: undefined reference to `__ubsan_handle_type_mismatch_v1'
./include/linux/compiler.h:254: undefined reference to `__ubsan_handle_type_mismatch_v1'
./include/linux/tracehook.h:190: undefined reference to `__ubsan_handle_type_mismatch_v1'
./include/linux/thread_info.h:60: undefined reference to `__ubsan_handle_type_mismatch_v1'
./include/trace/events/syscalls.h:17: undefined reference to `__ubsan_handle_type_mismatch_v1'
testing release v4.11
testing commit a351e9b9fc24e982ec2f0e76379a49826036da12 with gcc (GCC) 7.3.0
kernel signature: 6700388f894a6f7291b1f55101d150933df315d488d68509b4547027c5bacc03
run #0: basic kernel testing failed: UBSAN: undefined-behaviour in vhost_init_is_le
run #1: basic kernel testing failed: UBSAN: undefined-behaviour in vhost_init_is_le
run #2: basic kernel testing failed: UBSAN: undefined-behaviour in vhost_init_is_le
run #3: basic kernel testing failed: UBSAN: undefined-behaviour in vhost_init_is_le
run #4: basic kernel testing failed: UBSAN: undefined-behaviour in vhost_init_is_le
run #5: basic kernel testing failed: UBSAN: undefined-behaviour in vhost_init_is_le
run #6: basic kernel testing failed: UBSAN: undefined-behaviour in vhost_init_is_le
run #7: basic kernel testing failed: UBSAN: undefined-behaviour in vhost_init_is_le
run #8: basic kernel testing failed: UBSAN: undefined-behaviour in vhost_init_is_le
run #9: crashed: BUG: sleeping function called from invalid context in tap_get_minor
testing release v4.10
testing commit c470abd4fde40ea6a0846a2beab642a578c0b8cd with gcc (GCC) 5.5.0
kernel signature: 327c4f31943ac94ee5e133b6ae703066ca7a28280e497e20ef9006b4f5f628fa
run #0: basic kernel testing failed: UBSAN: undefined-behaviour in vhost_init_is_le
run #1: basic kernel testing failed: UBSAN: undefined-behaviour in vhost_init_is_le
run #2: basic kernel testing failed: UBSAN: undefined-behaviour in vhost_init_is_le
run #3: basic kernel testing failed: UBSAN: undefined-behaviour in vhost_init_is_le
run #4: basic kernel testing failed: UBSAN: undefined-behaviour in vhost_init_is_le
run #5: basic kernel testing failed: UBSAN: undefined-behaviour in vhost_init_is_le
run #6: basic kernel testing failed: UBSAN: undefined-behaviour in vhost_init_is_le
run #7: basic kernel testing failed: UBSAN: undefined-behaviour in vhost_init_is_le
run #8: basic kernel testing failed: UBSAN: undefined-behaviour in vhost_init_is_le
run #9: crashed: WARNING in sta_info_insert_rcu
testing release v4.9
testing commit 69973b830859bc6529a7a0468ba0d80ee5117826 with gcc (GCC) 5.5.0
kernel signature: 8b6c20f0a11ada402e603481fcd457e2351880ee30948d9e61b321b608224b51
all runs: crashed: WARNING in sta_info_insert_rcu
testing release v4.8
testing commit c8d2bc9bc39ebea8437fd974fdbc21847bb897a3 with gcc (GCC) 5.5.0
kernel signature: 30f74367bfb11296f080cfdedd4df5d2344e15c43635147516b19020b75a1d73
all runs: crashed: WARNING in sta_info_insert_rcu
testing release v4.7
testing commit 523d939ef98fd712632d93a5a2b588e477a7565e with gcc (GCC) 5.5.0
kernel signature: b683c4446406af32e6397142e5172895aee163830706622f66ebbb217ec9d66f
run #0: crashed: UBSAN: undefined-behaviour in ip_idents_reserve
run #1: crashed: UBSAN: undefined-behaviour in ip_idents_reserve
run #2: crashed: UBSAN: undefined-behaviour in ip_idents_reserve
run #3: crashed: UBSAN: undefined-behaviour in ip_idents_reserve
run #4: crashed: KASAN: null-ptr-deref Read

[dvyukov]

Another issue:
this also enabled UBSAN on LTS trees where UBSAN is not working well and now we have tons of UBSAN reports on LTS trees, e.g.:
https://syzkaller.appspot.com/bug?id=4f9547fee51db6b9d68e70bfc8f6f75ccda3d38b