kernel oops when running hip kernel with dev branch ROCR/ROCK #2

mattmacy · 2016-03-24T18:38:13Z

When running the square.cpp sample I now see the following:

Mar 24 11:28:34 pandemonium kernel: [ 639.895604] nvidia_uvm: Loaded the UVM driver, major device number 245
Mar 24 11:29:06 pandemonium kernel: [ 671.693636] amdgpu: vram aperture is out of 40bit address base: 0x383fc0000000 limit 0x383fd0000000
Mar 24 11:29:06 pandemonium kernel: [ 671.693749] amdgpu: vram aperture is out of 40bit address base: 0x383fe0000000 limit 0x383ff0000000
Mar 24 11:29:06 pandemonium kernel: [ 671.696239] amdgpu: vram aperture is out of 40bit address base: 0x383fc0000000 limit 0x383fd0000000
Mar 24 11:29:06 pandemonium kernel: [ 671.734321] amdgpu: vram aperture is out of 40bit address base: 0x383fe0000000 limit 0x383ff0000000
Mar 24 11:29:06 pandemonium kernel: [ 671.776858] BUG: unable to handle kernel paging request at ffffc90019ecd000
Mar 24 11:29:06 pandemonium kernel: [ 671.776863] IP: [] set_trap_handler+0x1a/0x30 [amdkfd]
Mar 24 11:29:06 pandemonium kernel: [ 671.776879] PGD ffec8f067 PUD ffeca0067 PMD fcf5f2067 PTE 0
Mar 24 11:29:06 pandemonium kernel: [ 671.776883] Oops: 0002 [#1] SMP
Mar 24 11:29:06 pandemonium kernel: [ 671.776886] Modules linked in: nvidia_uvm(POE) vmw_vsock_vmci_transport vsock vmw_vmci rfcomm bnep binfmt_misc hid_logitech_hidpp btusb btbcm btintel bluetooth b43 mac80211 nls_iso8859_1 cfg80211 ssb intel_rapl iosf_mbi x86_pkg_temp_thermal eeepc_wmi intel_powerclamp coretemp asus_wmi sparse_keymap video mxm_wmi kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel nvidia(POE) aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd serio_raw sb_edac edac_core snd_hda_codec_realtek snd_usb_audio snd_hda_codec_generic snd_usbmidi_lib snd_seq_midi hid_logitech_dj snd_seq_midi_event snd_hda_codec_hdmi snd_rawmidi snd_hda_intel snd_hda_controller snd_hda_codec snd_seq snd_hda_core snd_hwdep snd_seq_device bcma snd_pcm snd_timer snd mei_me lpc_ich soundcore mei shpchp wmi tpm_infineon mac_hid parport_pc ppdev lp parport autofs4 hid_generic usbhid hid amdkfd amd_iommu_v2 amdgpu psmouse amd_gnb_bus i2c_algo_bit e1000e ttm drm_kms_helper ahci ptp libahci drm pps_core
Mar 24 11:29:06 pandemonium kernel: [ 671.776940] CPU: 0 PID: 4040 Comm: a.out Tainted: P OE 4.1.0-201603162000-kfd-build-obsidian-82-generic #82
Mar 24 11:29:06 pandemonium kernel: [ 671.776943] Hardware name: iXsystems CSE-COR-AIR540/RAMPAGE V EXTREME, BIOS 1902 12/18/2015
Mar 24 11:29:06 pandemonium kernel: [ 671.776944] task: ffff880e71d93250 ti: ffff880ea4020000 task.ti: ffff880ea4020000
Mar 24 11:29:06 pandemonium kernel: [ 671.776946] RIP: 0010:[] [] set_trap_handler+0x1a/0x30 [amdkfd]
Mar 24 11:29:06 pandemonium kernel: [ 671.776955] RSP: 0018:ffff880ea4023d48 EFLAGS: 00010286
Mar 24 11:29:06 pandemonium kernel: [ 671.776956] RAX: ffffc90019ecd000 RBX: ffff880ff0f92e00 RCX: 0000000000000000
Mar 24 11:29:06 pandemonium kernel: [ 671.776957] RDX: 0000000002400000 RSI: ffff880ff3251e20 RDI: ffff880ff0f92a00
Mar 24 11:29:06 pandemonium kernel: [ 671.776959] RBP: ffff880ea4023d48 R08: ffff880ff0f92a00 R09: 0000000000000000
Mar 24 11:29:06 pandemonium kernel: [ 671.776960] R10: ffff880f962af800 R11: 00007ffc39269e80 R12: ffff880ea4023dc0
Mar 24 11:29:06 pandemonium kernel: [ 671.776961] R13: ffff880fb1275018 R14: ffff880fb1275000 R15: ffff880ea4023dc0
Mar 24 11:29:06 pandemonium kernel: [ 671.776963] FS: 00007f8005ebc740(0000) GS:ffff880fff200000(0000) knlGS:0000000000000000
Mar 24 11:29:06 pandemonium kernel: [ 671.776965] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 24 11:29:06 pandemonium kernel: [ 671.776966] CR2: ffffc90019ecd000 CR3: 0000000f3f50b000 CR4: 00000000001407f0
Mar 24 11:29:06 pandemonium kernel: [ 671.776968] Stack:
Mar 24 11:29:06 pandemonium kernel: [ 671.776969] ffff880ea4023d78 ffffffffc03d8883 ffff880ea4023dc0 fffffffffffffff2
Mar 24 11:29:06 pandemonium kernel: [ 671.776972] 000000000000001a 00000000fffffff2 ffff880ea4023e78 ffffffffc03d9ebf
Mar 24 11:29:06 pandemonium kernel: [ 671.776975] ffff880f962af800 ffffffffc03d8810 ffff880fb1275000 00007ffc39269e80
Mar 24 11:29:06 pandemonium kernel: [ 671.776977] Call Trace:
Mar 24 11:29:06 pandemonium kernel: [ 671.776986] [] kfd_ioctl_set_trap_handler+0x73/0xc0 [amdkfd]
Mar 24 11:29:06 pandemonium kernel: [ 671.776994] [] kfd_ioctl+0x2bf/0x4d0 [amdkfd]
Mar 24 11:29:06 pandemonium kernel: [ 671.777001] [] ? kfd_ioctl_get_process_apertures+0x2e0/0x2e0 [amdkfd]
Mar 24 11:29:06 pandemonium kernel: [ 671.777010] [] ? pte_alloc_one+0x30/0x50
Mar 24 11:29:06 pandemonium kernel: [ 671.777015] [] ? __pte_alloc+0xcc/0x180
Mar 24 11:29:06 pandemonium kernel: [ 671.777019] [] do_vfs_ioctl+0x2f8/0x510
Mar 24 11:29:06 pandemonium kernel: [ 671.777023] [] ? __do_page_fault+0x1b6/0x450
Mar 24 11:29:06 pandemonium kernel: [ 671.777026] [] SyS_ioctl+0x81/0xa0
Mar 24 11:29:06 pandemonium kernel: [ 671.777028] [] ? do_page_fault+0x30/0x80
Mar 24 11:29:06 pandemonium kernel: [ 671.777032] [] system_call_fastpath+0x16/0x75
Mar 24 11:29:06 pandemonium kernel: [ 671.777034] Code: 00 0f 1f 44 00 00 55 31 c0 48 89 e5 5d c3 0f 1f 00 0f 1f 44 00 00 55 48 8b 46 f0 48 89 e5 8b 80 f4 01 00 00 48 03 86 e0 00 00 00 <48> 89 10 48 89 48 08 31 c0 5d c3 66 66 2e 0f 1f 84 00 00 00 00
Mar 24 11:29:06 pandemonium kernel: [ 671.777061] RIP [] set_trap_handler+0x1a/0x30 [amdkfd]
Mar 24 11:29:06 pandemonium kernel: [ 671.777068] RSP
Mar 24 11:29:06 pandemonium kernel: [ 671.777069] CR2: ffffc90019ecd000
Mar 24 11:29:06 pandemonium kernel: [ 671.777072] ---[ end trace 53807749a7eb2ed3 ]---

jedwards-AMD · 2016-04-04T19:51:54Z

This should be fixed with the new dev branch.

mattmacy · 2016-04-25T18:20:37Z

Working on the BSD port. Assuming fixed.

…uct_mutex This patch fixes lockdep issue due to circular locking dependency of struct_mutex, i_mutex_key, mmap_sem, relay_channels_mutex. For GuC log relay channel we create debugfs file that requires i_mutex_key lock and we are doing that under struct_mutex. So we introduced newer dependency as: &dev->struct_mutex --> &sb->s_type->i_mutex_key#3 --> &mm->mmap_sem However, there is dependency from mmap_sem to struct_mutex. Hence we separate the relay create/destroy operation from under struct_mutex. Also added runtime check of relay buffer status. Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk> ====================================================== WARNING: possible circular locking dependency detected 4.15.0-rc6-CI-Patchwork_7614+ #1 Not tainted ------------------------------------------------------ debugfs_test/1388 is trying to acquire lock: (&dev->struct_mutex){+.+.}, at: [<00000000d5e1d915>] i915_mutex_lock_interruptible+0x47/0x130 [i915] but task is already holding lock: (&mm->mmap_sem){++++}, at: [<0000000029a9c131>] __do_page_fault+0x106/0x560 which lock already depends on the new lock. the existing dependency chain (in reverse order) is: -> #3 (&mm->mmap_sem){++++}: _copy_to_user+0x1e/0x70 filldir+0x8c/0xf0 dcache_readdir+0xeb/0x160 iterate_dir+0xdc/0x140 SyS_getdents+0xa0/0x130 entry_SYSCALL_64_fastpath+0x1c/0x89 -> #2 (&sb->s_type->i_mutex_key#3){++++}: start_creating+0x59/0x110 __debugfs_create_file+0x2e/0xe0 relay_create_buf_file+0x62/0x80 relay_late_setup_files+0x84/0x250 guc_log_late_setup+0x4f/0x110 [i915] i915_guc_log_register+0x32/0x40 [i915] i915_driver_load+0x7b6/0x1720 [i915] i915_pci_probe+0x2e/0x90 [i915] pci_device_probe+0x9c/0x120 driver_probe_device+0x2a3/0x480 __driver_attach+0xd9/0xe0 bus_for_each_dev+0x57/0x90 bus_add_driver+0x168/0x260 driver_register+0x52/0xc0 do_one_initcall+0x39/0x150 do_init_module+0x56/0x1ef load_module+0x231c/0x2d70 SyS_finit_module+0xa5/0xe0 entry_SYSCALL_64_fastpath+0x1c/0x89 -> #1 (relay_channels_mutex){+.+.}: relay_open+0x12c/0x2b0 intel_guc_log_runtime_create+0xab/0x230 [i915] intel_guc_init+0x81/0x120 [i915] intel_uc_init+0x29/0xa0 [i915] i915_gem_init+0x182/0x530 [i915] i915_driver_load+0xaa9/0x1720 [i915] i915_pci_probe+0x2e/0x90 [i915] pci_device_probe+0x9c/0x120 driver_probe_device+0x2a3/0x480 __driver_attach+0xd9/0xe0 bus_for_each_dev+0x57/0x90 bus_add_driver+0x168/0x260 driver_register+0x52/0xc0 do_one_initcall+0x39/0x150 do_init_module+0x56/0x1ef load_module+0x231c/0x2d70 SyS_finit_module+0xa5/0xe0 entry_SYSCALL_64_fastpath+0x1c/0x89 -> #0 (&dev->struct_mutex){+.+.}: __mutex_lock+0x81/0x9b0 i915_mutex_lock_interruptible+0x47/0x130 [i915] i915_gem_fault+0x201/0x790 [i915] __do_fault+0x15/0x70 __handle_mm_fault+0x677/0xdc0 handle_mm_fault+0x14f/0x2f0 __do_page_fault+0x2d1/0x560 page_fault+0x4c/0x60 other info that might help us debug this: Chain exists of: &dev->struct_mutex --> &sb->s_type->i_mutex_key#3 --> &mm->mmap_sem Possible unsafe locking scenario: CPU0 CPU1 ---- ---- lock(&mm->mmap_sem); lock(&sb->s_type->i_mutex_key#3); lock(&mm->mmap_sem); lock(&dev->struct_mutex); *** DEADLOCK *** 1 lock held by debugfs_test/1388: #0: (&mm->mmap_sem){++++}, at: [<0000000029a9c131>] __do_page_fault+0x106/0x560 stack backtrace: CPU: 2 PID: 1388 Comm: debugfs_test Not tainted 4.15.0-rc6-CI-Patchwork_7614+ #1 Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./J4205-ITX, BIOS P1.10 09/29/2016 Call Trace: dump_stack+0x5f/0x86 print_circular_bug.isra.18+0x1d0/0x2c0 __lock_acquire+0x14ae/0x1b60 ? lock_acquire+0xaf/0x200 lock_acquire+0xaf/0x200 ? i915_mutex_lock_interruptible+0x47/0x130 [i915] __mutex_lock+0x81/0x9b0 ? i915_mutex_lock_interruptible+0x47/0x130 [i915] ? i915_mutex_lock_interruptible+0x47/0x130 [i915] ? i915_mutex_lock_interruptible+0x47/0x130 [i915] i915_mutex_lock_interruptible+0x47/0x130 [i915] ? __pm_runtime_resume+0x4f/0x80 i915_gem_fault+0x201/0x790 [i915] __do_fault+0x15/0x70 ? _raw_spin_unlock+0x29/0x40 __handle_mm_fault+0x677/0xdc0 handle_mm_fault+0x14f/0x2f0 __do_page_fault+0x2d1/0x560 ? page_fault+0x36/0x60 page_fault+0x4c/0x60 v2: Added lock protection to guc->log.runtime.relay_chan (Chris) Fixed locking inside guc_flush_logs uncovered by new lockdep. v3: Locking guc_read_update_log_buffer entirely with relay_lock. (Chris) Prepared intel_guc_init_early. Moved relay_lock inside relay_create relay_destroy, relay_file_create, guc_read_update_log_buffer. (Michal) Removed struct_mutex lock around guc_log_flush and removed usage of guc_log_has_relay() from runtime_create path as it needs struct_mutex lock. v4: Handle NULL relay sub buffer pointer earlier in read_update_log_buffer (Chris). Fixed comment suffix **/. (Michal) Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=104693 Testcase: igt/debugfs_test/read_all_entries # with enable_guc=1 and guc_log_level=1 Signed-off-by: Sagar Arun Kamble <sagar.a.kamble@intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Marta Lofstedt <marta.lofstedt@intel.com> Cc: Michal Winiarski <michal.winiarski@intel.com> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Link: https://patchwork.freedesktop.org/patch/msgid/1516808821-3638-3-git-send-email-sagar.a.kamble@intel.com

Heiner reported a lockdep splat [1] This is caused by attempting GFP_KERNEL allocation while RCU lock is held and BH blocked. We believe that addrconf_verify_rtnl() could run for a long period, so instead of using GFP_ATOMIC here as Ido suggested, we should break the critical section and restart it after the allocation. [1] [86220.125562] ============================= [86220.125586] WARNING: suspicious RCU usage [86220.125612] 4.15.0-rc7-next-20180110+ #7 Not tainted [86220.125641] ----------------------------- [86220.125666] kernel/sched/core.c:6026 Illegal context switch in RCU-bh read-side critical section! [86220.125711] other info that might help us debug this: [86220.125755] rcu_scheduler_active = 2, debug_locks = 1 [86220.125792] 4 locks held by kworker/0:2/1003: [86220.125817] #0: ((wq_completion)"%s"("ipv6_addrconf")){+.+.}, at: [<00000000da8e9b73>] process_one_work+0x1de/0x680 [86220.125895] #1: ((addr_chk_work).work){+.+.}, at: [<00000000da8e9b73>] process_one_work+0x1de/0x680 [86220.125959] #2: (rtnl_mutex){+.+.}, at: [<00000000b06d9510>] rtnl_lock+0x12/0x20 [86220.126017] #3: (rcu_read_lock_bh){....}, at: [<00000000aef52299>] addrconf_verify_rtnl+0x1e/0x510 [ipv6] [86220.126111] stack backtrace: [86220.126142] CPU: 0 PID: 1003 Comm: kworker/0:2 Not tainted 4.15.0-rc7-next-20180110+ #7 [86220.126185] Hardware name: ZOTAC ZBOX-CI321NANO/ZBOX-CI321NANO, BIOS B246P105 06/01/2015 [86220.126250] Workqueue: ipv6_addrconf addrconf_verify_work [ipv6] [86220.126288] Call Trace: [86220.126312] dump_stack+0x70/0x9e [86220.126337] lockdep_rcu_suspicious+0xce/0xf0 [86220.126365] ___might_sleep+0x1d3/0x240 [86220.126390] __might_sleep+0x45/0x80 [86220.126416] kmem_cache_alloc_trace+0x53/0x250 [86220.126458] ? ipv6_add_addr+0xfe/0x6e0 [ipv6] [86220.126498] ipv6_add_addr+0xfe/0x6e0 [ipv6] [86220.126538] ipv6_create_tempaddr+0x24d/0x430 [ipv6] [86220.126580] ? ipv6_create_tempaddr+0x24d/0x430 [ipv6] [86220.126623] addrconf_verify_rtnl+0x339/0x510 [ipv6] [86220.126664] ? addrconf_verify_rtnl+0x339/0x510 [ipv6] [86220.126708] addrconf_verify_work+0xe/0x20 [ipv6] [86220.126738] process_one_work+0x258/0x680 [86220.126765] worker_thread+0x35/0x3f0 [86220.126790] kthread+0x124/0x140 [86220.126813] ? process_one_work+0x680/0x680 [86220.126839] ? kthread_create_worker_on_cpu+0x40/0x40 [86220.126869] ? umh_complete+0x40/0x40 [86220.126893] ? call_usermodehelper_exec_async+0x12a/0x160 [86220.126926] ret_from_fork+0x4b/0x60 [86220.126999] BUG: sleeping function called from invalid context at mm/slab.h:420 [86220.127041] in_atomic(): 1, irqs_disabled(): 0, pid: 1003, name: kworker/0:2 [86220.127082] 4 locks held by kworker/0:2/1003: [86220.127107] #0: ((wq_completion)"%s"("ipv6_addrconf")){+.+.}, at: [<00000000da8e9b73>] process_one_work+0x1de/0x680 [86220.127179] #1: ((addr_chk_work).work){+.+.}, at: [<00000000da8e9b73>] process_one_work+0x1de/0x680 [86220.127242] #2: (rtnl_mutex){+.+.}, at: [<00000000b06d9510>] rtnl_lock+0x12/0x20 [86220.127300] #3: (rcu_read_lock_bh){....}, at: [<00000000aef52299>] addrconf_verify_rtnl+0x1e/0x510 [ipv6] [86220.127414] CPU: 0 PID: 1003 Comm: kworker/0:2 Not tainted 4.15.0-rc7-next-20180110+ #7 [86220.127463] Hardware name: ZOTAC ZBOX-CI321NANO/ZBOX-CI321NANO, BIOS B246P105 06/01/2015 [86220.127528] Workqueue: ipv6_addrconf addrconf_verify_work [ipv6] [86220.127568] Call Trace: [86220.127591] dump_stack+0x70/0x9e [86220.127616] ___might_sleep+0x14d/0x240 [86220.127644] __might_sleep+0x45/0x80 [86220.127672] kmem_cache_alloc_trace+0x53/0x250 [86220.127717] ? ipv6_add_addr+0xfe/0x6e0 [ipv6] [86220.127762] ipv6_add_addr+0xfe/0x6e0 [ipv6] [86220.127807] ipv6_create_tempaddr+0x24d/0x430 [ipv6] [86220.127854] ? ipv6_create_tempaddr+0x24d/0x430 [ipv6] [86220.127903] addrconf_verify_rtnl+0x339/0x510 [ipv6] [86220.127950] ? addrconf_verify_rtnl+0x339/0x510 [ipv6] [86220.127998] addrconf_verify_work+0xe/0x20 [ipv6] [86220.128032] process_one_work+0x258/0x680 [86220.128063] worker_thread+0x35/0x3f0 [86220.128091] kthread+0x124/0x140 [86220.128117] ? process_one_work+0x680/0x680 [86220.128146] ? kthread_create_worker_on_cpu+0x40/0x40 [86220.128180] ? umh_complete+0x40/0x40 [86220.128207] ? call_usermodehelper_exec_async+0x12a/0x160 [86220.128243] ret_from_fork+0x4b/0x60 Fixes: f3d9832 ("ipv6: addrconf: cleanup locking in ipv6_add_addr") Signed-off-by: Eric Dumazet <edumazet@google.com> Reported-by: Heiner Kallweit <hkallweit1@gmail.com> Reviewed-by: Ido Schimmel <idosch@mellanox.com> Signed-off-by: David S. Miller <davem@davemloft.net>

The following soft lockup was caught. This is a deadlock caused by recusive locking. Process kworker/u40:1:28016 was holding spin lock "mbx->queue_lock" in qlcnic_83xx_mailbox_worker(), while a softirq came in and ask the same spin lock in qlcnic_83xx_enqueue_mbx_cmd(). This lock should be hold by disable bh.. [161846.962125] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u40:1:28016] [161846.962367] Modules linked in: tun ocfs2 xen_netback xen_blkback xen_gntalloc xen_gntdev xen_evtchn xenfs xen_privcmd autofs4 ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue configfs bnx2fc fcoe libfcoe libfc sunrpc 8021q mrp garp bridge stp llc bonding dm_round_robin dm_multipath iTCO_wdt iTCO_vendor_support pcspkr sb_edac edac_core i2c_i801 shpchp lpc_ich mfd_core ioatdma ipmi_devintf ipmi_si ipmi_msghandler sg ext4 jbd2 mbcache2 sr_mod cdrom sd_mod igb i2c_algo_bit i2c_core ahci libahci megaraid_sas ixgbe dca ptp pps_core vxlan udp_tunnel ip6_udp_tunnel qla2xxx scsi_transport_fc qlcnic crc32c_intel be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi ipv6 cxgb3 mdio libiscsi_tcp qla4xxx iscsi_boot_sysfs libiscsi scsi_transport_iscsi dm_mirror dm_region_hash dm_log dm_mod [161846.962454] [161846.962460] CPU: 1 PID: 28016 Comm: kworker/u40:1 Not tainted 4.1.12-94.5.9.el6uek.x86_64 #2 [161846.962463] Hardware name: Oracle Corporation SUN SERVER X4-2L /ASSY,MB,X4-2L , BIOS 26050100 09/19/2017 [161846.962489] Workqueue: qlcnic_mailbox qlcnic_83xx_mailbox_worker [qlcnic] [161846.962493] task: ffff8801f2e34600 ti: ffff88004ca5c000 task.ti: ffff88004ca5c000 [161846.962496] RIP: e030:[<ffffffff810013aa>] [<ffffffff810013aa>] xen_hypercall_sched_op+0xa/0x20 [161846.962506] RSP: e02b:ffff880202e43388 EFLAGS: 00000206 [161846.962509] RAX: 0000000000000000 RBX: ffff8801f6996b70 RCX: ffffffff810013aa [161846.962511] RDX: ffff880202e433cc RSI: ffff880202e433b0 RDI: 0000000000000003 [161846.962513] RBP: ffff880202e433d0 R08: 0000000000000000 R09: ffff8801fe893200 [161846.962516] R10: ffff8801fe400538 R11: 0000000000000206 R12: ffff880202e4b000 [161846.962518] R13: 0000000000000050 R14: 0000000000000001 R15: 000000000000020d [161846.962528] FS: 0000000000000000(0000) GS:ffff880202e40000(0000) knlGS:ffff880202e40000 [161846.962531] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 [161846.962533] CR2: 0000000002612640 CR3: 00000001bb796000 CR4: 0000000000042660 [161846.962536] Stack: [161846.962538] ffff880202e43608 0000000000000000 ffffffff813f0442 ffff880202e433b0 [161846.962543] 0000000000000000 ffff880202e433cc ffffffff00000001 0000000000000000 [161846.962547] 00000009813f03d6 ffff880202e433e0 ffffffff813f0460 ffff880202e43440 [161846.962552] Call Trace: [161846.962555] <IRQ> [161846.962565] [<ffffffff813f0442>] ? xen_poll_irq_timeout+0x42/0x50 [161846.962570] [<ffffffff813f0460>] xen_poll_irq+0x10/0x20 [161846.962578] [<ffffffff81014222>] xen_lock_spinning+0xe2/0x110 [161846.962583] [<ffffffff81013f01>] __raw_callee_save_xen_lock_spinning+0x11/0x20 [161846.962592] [<ffffffff816e5c57>] ? _raw_spin_lock+0x57/0x80 [161846.962609] [<ffffffffa028acfc>] qlcnic_83xx_enqueue_mbx_cmd+0x7c/0xe0 [qlcnic] [161846.962623] [<ffffffffa028e008>] qlcnic_83xx_issue_cmd+0x58/0x210 [qlcnic] [161846.962636] [<ffffffffa028caf2>] qlcnic_83xx_sre_macaddr_change+0x162/0x1d0 [qlcnic] [161846.962649] [<ffffffffa028cb8b>] qlcnic_83xx_change_l2_filter+0x2b/0x30 [qlcnic] [161846.962657] [<ffffffff8160248b>] ? __skb_flow_dissect+0x18b/0x650 [161846.962670] [<ffffffffa02856e5>] qlcnic_send_filter+0x205/0x250 [qlcnic] [161846.962682] [<ffffffffa0285c77>] qlcnic_xmit_frame+0x547/0x7b0 [qlcnic] [161846.962691] [<ffffffff8160ac22>] xmit_one+0x82/0x1a0 [161846.962696] [<ffffffff8160ad90>] dev_hard_start_xmit+0x50/0xa0 [161846.962701] [<ffffffff81630112>] sch_direct_xmit+0x112/0x220 [161846.962706] [<ffffffff8160b80f>] __dev_queue_xmit+0x1df/0x5e0 [161846.962710] [<ffffffff8160bc33>] dev_queue_xmit_sk+0x13/0x20 [161846.962721] [<ffffffffa0575bd5>] bond_dev_queue_xmit+0x35/0x80 [bonding] [161846.962729] [<ffffffffa05769fb>] __bond_start_xmit+0x1cb/0x210 [bonding] [161846.962736] [<ffffffffa0576a71>] bond_start_xmit+0x31/0x60 [bonding] [161846.962740] [<ffffffff8160ac22>] xmit_one+0x82/0x1a0 [161846.962745] [<ffffffff8160ad90>] dev_hard_start_xmit+0x50/0xa0 [161846.962749] [<ffffffff8160bb1e>] __dev_queue_xmit+0x4ee/0x5e0 [161846.962754] [<ffffffff8160bc33>] dev_queue_xmit_sk+0x13/0x20 [161846.962760] [<ffffffffa05cfa72>] vlan_dev_hard_start_xmit+0xb2/0x150 [8021q] [161846.962764] [<ffffffff8160ac22>] xmit_one+0x82/0x1a0 [161846.962769] [<ffffffff8160ad90>] dev_hard_start_xmit+0x50/0xa0 [161846.962773] [<ffffffff8160bb1e>] __dev_queue_xmit+0x4ee/0x5e0 [161846.962777] [<ffffffff8160bc33>] dev_queue_xmit_sk+0x13/0x20 [161846.962789] [<ffffffffa05adf74>] br_dev_queue_push_xmit+0x54/0xa0 [bridge] [161846.962797] [<ffffffffa05ae4ff>] br_forward_finish+0x2f/0x90 [bridge] [161846.962807] [<ffffffff810b0dad>] ? ttwu_do_wakeup+0x1d/0x100 [161846.962811] [<ffffffff815f929b>] ? __alloc_skb+0x8b/0x1f0 [161846.962818] [<ffffffffa05ae04d>] __br_forward+0x8d/0x120 [bridge] [161846.962822] [<ffffffff815f613b>] ? __kmalloc_reserve+0x3b/0xa0 [161846.962829] [<ffffffff810be55e>] ? update_rq_runnable_avg+0xee/0x230 [161846.962836] [<ffffffffa05ae176>] br_forward+0x96/0xb0 [bridge] [161846.962845] [<ffffffffa05af85e>] br_handle_frame_finish+0x1ae/0x420 [bridge] [161846.962853] [<ffffffffa05afc4f>] br_handle_frame+0x17f/0x260 [bridge] [161846.962862] [<ffffffffa05afad0>] ? br_handle_frame_finish+0x420/0x420 [bridge] [161846.962867] [<ffffffff8160d057>] __netif_receive_skb_core+0x1f7/0x870 [161846.962872] [<ffffffff8160d6f2>] __netif_receive_skb+0x22/0x70 [161846.962877] [<ffffffff8160d913>] netif_receive_skb_internal+0x23/0x90 [161846.962884] [<ffffffffa07512ea>] ? xenvif_idx_release+0xea/0x100 [xen_netback] [161846.962889] [<ffffffff816e5a10>] ? _raw_spin_unlock_irqrestore+0x20/0x50 [161846.962893] [<ffffffff8160e624>] netif_receive_skb_sk+0x24/0x90 [161846.962899] [<ffffffffa075269a>] xenvif_tx_submit+0x2ca/0x3f0 [xen_netback] [161846.962906] [<ffffffffa0753f0c>] xenvif_tx_action+0x9c/0xd0 [xen_netback] [161846.962915] [<ffffffffa07567f5>] xenvif_poll+0x35/0x70 [xen_netback] [161846.962920] [<ffffffff8160e01b>] napi_poll+0xcb/0x1e0 [161846.962925] [<ffffffff8160e1c0>] net_rx_action+0x90/0x1c0 [161846.962931] [<ffffffff8108aaba>] __do_softirq+0x10a/0x350 [161846.962938] [<ffffffff8108ae75>] irq_exit+0x125/0x130 [161846.962943] [<ffffffff813f03a9>] xen_evtchn_do_upcall+0x39/0x50 [161846.962950] [<ffffffff816e7ffe>] xen_do_hypervisor_callback+0x1e/0x40 [161846.962952] <EOI> [161846.962959] [<ffffffff816e5c4a>] ? _raw_spin_lock+0x4a/0x80 [161846.962964] [<ffffffff816e5b1e>] ? _raw_spin_lock_irqsave+0x1e/0xa0 [161846.962978] [<ffffffffa028e279>] ? qlcnic_83xx_mailbox_worker+0xb9/0x2a0 [qlcnic] [161846.962991] [<ffffffff810a14e1>] ? process_one_work+0x151/0x4b0 [161846.962995] [<ffffffff8100c3f2>] ? check_events+0x12/0x20 [161846.963001] [<ffffffff810a1960>] ? worker_thread+0x120/0x480 [161846.963005] [<ffffffff816e187b>] ? __schedule+0x30b/0x890 [161846.963010] [<ffffffff810a1840>] ? process_one_work+0x4b0/0x4b0 [161846.963015] [<ffffffff810a1840>] ? process_one_work+0x4b0/0x4b0 [161846.963021] [<ffffffff810a6b3e>] ? kthread+0xce/0xf0 [161846.963025] [<ffffffff810a6a70>] ? kthread_freezable_should_stop+0x70/0x70 [161846.963031] [<ffffffff816e6522>] ? ret_from_fork+0x42/0x70 [161846.963035] [<ffffffff810a6a70>] ? kthread_freezable_should_stop+0x70/0x70 [161846.963037] Code: cc 51 41 53 b8 1c 00 00 00 0f 05 41 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 1d 00 00 00 0f 05 <41> 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc Signed-off-by: Junxiao Bi <junxiao.bi@oracle.com> Signed-off-by: David S. Miller <davem@davemloft.net>

Newly added igmpv3_get_srcaddr() needs to be called under rcu lock. Timer callbacks do not ensure this locking. ============================= WARNING: suspicious RCU usage 4.15.0+ #200 Not tainted ----------------------------- ./include/linux/inetdevice.h:216 suspicious rcu_dereference_check() usage! other info that might help us debug this: rcu_scheduler_active = 2, debug_locks = 1 3 locks held by syzkaller616973/4074: #0: (&mm->mmap_sem){++++}, at: [<00000000bfce669e>] __do_page_fault+0x32d/0xc90 arch/x86/mm/fault.c:1355 #1: ((&im->timer)){+.-.}, at: [<00000000619d2f71>] lockdep_copy_map include/linux/lockdep.h:178 [inline] #1: ((&im->timer)){+.-.}, at: [<00000000619d2f71>] call_timer_fn+0x1c6/0x820 kernel/time/timer.c:1316 #2: (&(&im->lock)->rlock){+.-.}, at: [<000000005f833c5c>] spin_lock_bh include/linux/spinlock.h:315 [inline] #2: (&(&im->lock)->rlock){+.-.}, at: [<000000005f833c5c>] igmpv3_send_report+0x98/0x5b0 net/ipv4/igmp.c:600 stack backtrace: CPU: 0 PID: 4074 Comm: syzkaller616973 Not tainted 4.15.0+ #200 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: <IRQ> __dump_stack lib/dump_stack.c:17 [inline] dump_stack+0x194/0x257 lib/dump_stack.c:53 lockdep_rcu_suspicious+0x123/0x170 kernel/locking/lockdep.c:4592 __in_dev_get_rcu include/linux/inetdevice.h:216 [inline] igmpv3_get_srcaddr net/ipv4/igmp.c:329 [inline] igmpv3_newpack+0xeef/0x12e0 net/ipv4/igmp.c:389 add_grhead.isra.27+0x235/0x300 net/ipv4/igmp.c:432 add_grec+0xbd3/0x1170 net/ipv4/igmp.c:565 igmpv3_send_report+0xd5/0x5b0 net/ipv4/igmp.c:605 igmp_send_report+0xc43/0x1050 net/ipv4/igmp.c:722 igmp_timer_expire+0x322/0x5c0 net/ipv4/igmp.c:831 call_timer_fn+0x228/0x820 kernel/time/timer.c:1326 expire_timers kernel/time/timer.c:1363 [inline] __run_timers+0x7ee/0xb70 kernel/time/timer.c:1666 run_timer_softirq+0x4c/0x70 kernel/time/timer.c:1692 __do_softirq+0x2d7/0xb85 kernel/softirq.c:285 invoke_softirq kernel/softirq.c:365 [inline] irq_exit+0x1cc/0x200 kernel/softirq.c:405 exiting_irq arch/x86/include/asm/apic.h:541 [inline] smp_apic_timer_interrupt+0x16b/0x700 arch/x86/kernel/apic/apic.c:1052 apic_timer_interrupt+0xa9/0xb0 arch/x86/entry/entry_64.S:938 Fixes: a46182b ("net: igmp: Use correct source address on IGMPv3 reports") Signed-off-by: Eric Dumazet <edumazet@google.com> Reported-by: syzbot <syzkaller@googlegroups.com> Signed-off-by: David S. Miller <davem@davemloft.net>

Scenario: 1. Port down and do fail over 2. Ap do rds_bind syscall PID: 47039 TASK: ffff89887e2fe640 CPU: 47 COMMAND: "kworker/u:6" #0 [ffff898e35f159f0] machine_kexec at ffffffff8103abf9 #1 [ffff898e35f15a60] crash_kexec at ffffffff810b96e3 #2 [ffff898e35f15b30] oops_end at ffffffff8150f518 #3 [ffff898e35f15b60] no_context at ffffffff8104854c #4 [ffff898e35f15ba0] __bad_area_nosemaphore at ffffffff81048675 #5 [ffff898e35f15bf0] bad_area_nosemaphore at ffffffff810487d3 #6 [ffff898e35f15c00] do_page_fault at ffffffff815120b8 #7 [ffff898e35f15d10] page_fault at ffffffff8150ea95 [exception RIP: unknown or invalid address] RIP: 0000000000000000 RSP: ffff898e35f15dc8 RFLAGS: 00010282 RAX: 00000000fffffffe RBX: ffff889b77f6fc00 RCX:ffffffff81c99d88 RDX: 0000000000000000 RSI: ffff896019ee08e8 RDI:ffff889b77f6fc00 RBP: ffff898e35f15df0 R8: ffff896019ee08c8 R9:0000000000000000 R10: 0000000000000400 R11: 0000000000000000 R12:ffff896019ee08c0 R13: ffff889b77f6fe68 R14: ffffffff81c99d80 R15: ffffffffa022a1e0 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 #8 [ffff898e35f15dc8] cma_ndev_work_handler at ffffffffa022a228 [rdma_cm] #9 [ffff898e35f15df8] process_one_work at ffffffff8108a7c6 #10 [ffff898e35f15e58] worker_thread at ffffffff8108bda0 #11 [ffff898e35f15ee8] kthread at ffffffff81090fe6 PID: 45659 TASK: ffff880d313d2500 CPU: 31 COMMAND: "oracle_45659_ap" #0 [ffff881024ccfc98] __schedule at ffffffff8150bac4 #1 [ffff881024ccfd40] schedule at ffffffff8150c2cf #2 [ffff881024ccfd50] __mutex_lock_slowpath at ffffffff8150cee7 #3 [ffff881024ccfdc0] mutex_lock at ffffffff8150cdeb #4 [ffff881024ccfde0] rdma_destroy_id at ffffffffa022a027 [rdma_cm] #5 [ffff881024ccfe10] rds_ib_laddr_check at ffffffffa0357857 [rds_rdma] #6 [ffff881024ccfe50] rds_trans_get_preferred at ffffffffa0324c2a [rds] #7 [ffff881024ccfe80] rds_bind at ffffffffa031d690 [rds] #8 [ffff881024ccfeb0] sys_bind at ffffffff8142a670 PID: 45659 PID: 47039 rds_ib_laddr_check /* create id_priv with a null event_handler */ rdma_create_id rdma_bind_addr cma_acquire_dev /* add id_priv to cma_dev->id_list */ cma_attach_to_dev cma_ndev_work_handler /* event_hanlder is null */ id_priv->id.event_handler Signed-off-by: Guanglei Li <guanglei.li@oracle.com> Signed-off-by: Honglei Wang <honglei.wang@oracle.com> Reviewed-by: Junxiao Bi <junxiao.bi@oracle.com> Reviewed-by: Yanjun Zhu <yanjun.zhu@oracle.com> Reviewed-by: Leon Romanovsky <leonro@mellanox.com> Acked-by: Santosh Shilimkar <santosh.shilimkar@oracle.com> Acked-by: Doug Ledford <dledford@redhat.com> Signed-off-by: David S. Miller <davem@davemloft.net>

…nna/linux-nfs NFS-over-RDMA client fixes for Linux 4.16 #2 Stable fixes: - Fix calculating ri_max_send_sges, which can oops if max_sge is too small - Fix a BUG after device removal if freed resources haven't been allocated yet

Calling __UDPX_INC_STATS() from a preemptible context leads to a warning of the form: BUG: using __this_cpu_add() in preemptible [00000000] code: kworker/u5:0/31 caller is xs_udp_data_receive_workfn+0x194/0x270 CPU: 1 PID: 31 Comm: kworker/u5:0 Not tainted 4.15.0-rc8-00076-g90ea9f1 #2 Workqueue: xprtiod xs_udp_data_receive_workfn Call Trace: dump_stack+0x85/0xc1 check_preemption_disabled+0xce/0xe0 xs_udp_data_receive_workfn+0x194/0x270 process_one_work+0x318/0x620 worker_thread+0x20a/0x390 ? process_one_work+0x620/0x620 kthread+0x120/0x130 ? __kthread_bind_mask+0x60/0x60 ret_from_fork+0x24/0x30 Since we're taking a spinlock in those functions anyway, let's fix the issue by moving the call so that it occurs under the spinlock. Reported-by: kernel test robot <fengguang.wu@intel.com> Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>

Although we protect the request itself, we don't lock inside intel_engine_dump() and so the request maybe retired as we peek into it. One consequence is that the request->ctx may be freed before we dereference it, leading to a use-after-free. Replace the hw_id we are peeking from inside request->ctx with the request->fence.context, with which we can still track from which context the request originated (although to tie to HW reports requires a little more legwork, but is good enough to follow the GEM traces). [52640.729670] general protection fault: 0000 [#2] SMP [52640.729694] Dumping ftrace buffer: [52640.729701] (ftrace buffer empty) [52640.729705] Modules linked in: vgem snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic x86_pkg_\ temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul snd_hda_intel snd_hda_codec snd_hwdep gha\ sh_clmulni_intel snd_hda_core snd_pcm mei_me mei i915 r8169 mii prime_numbers i2c_hid [52640.729748] CPU: 2 PID: 4335 Comm: gem_exec_schedu Tainted: G UD W 4.16.0-rc3+ #7 [52640.729759] Hardware name: Acer Aspire E5-575G/Ironman_SK , BIOS V1.12 08/02/2016 [52640.729803] RIP: 0010:print_request+0x2b/0xb0 [i915] [52640.729811] RSP: 0018:ffffc90001453c18 EFLAGS: 00010206 [52640.729820] RAX: 6b6b6b6b6b6b6b6b RBX: ffff8801e0292d40 RCX: 0000000000000006 [52640.729829] RDX: ffffc90001453c60 RSI: ffff8801e0292d40 RDI: 0000000000000003 [52640.729838] RBP: ffffc90001453d80 R08: 0000000000000000 R09: 0000000000000001 [52640.729847] R10: ffffc90001453bd0 R11: ffffc90001453c73 R12: ffffc90001453c60 [52640.729856] R13: ffffc90001453d80 R14: ffff8801d5a683c8 R15: ffff8801e0292d40 [52640.729866] FS: 00007f1ee50548c0(0000) GS:ffff8801e8200000(0000) knlGS:0000000000000000 [52640.729876] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [52640.729884] CR2: 00007f1ee5077000 CR3: 00000001d9411004 CR4: 00000000003606e0 [52640.729893] Call Trace: [52640.729922] intel_engine_print_registers+0x623/0x890 [i915] [52640.729948] intel_engine_dump+0x4a3/0x590 [i915] [52640.729957] ? seq_printf+0x3a/0x50 [52640.729977] i915_engine_info+0xb8/0xe0 [i915] [52640.729984] ? drm_mode_gamma_get_ioctl+0xf0/0xf0 [52640.729990] seq_read+0xd5/0x410 [52640.729997] full_proxy_read+0x4b/0x70 [52640.730004] __vfs_read+0x1e/0x120 [52640.730009] ? do_sys_open+0x134/0x220 [52640.730015] ? kmem_cache_free+0x174/0x2b0 [52640.730021] vfs_read+0xa1/0x150 [52640.730026] SyS_read+0x40/0xa0 [52640.730032] do_syscall_64+0x65/0x1a0 [52640.730038] entry_SYSCALL_64_after_hwframe+0x42/0xb7 Reported-by: Mika Kuoppala <mika.kuoppala@intel.com> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Cc: Mika Kuoppala <mika.kuoppala@intel.com> Reviewed-by: Mika Kuoppala <mika.kuoppala@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20180228094732.28462-1-chris@chris-wilson.co.uk

The dead circular lock senario captured is as followed. The idea of the fix is moving read_user_wptr outside of acquire_queue...release_queue critical section [ 63.477482] WARNING: possible circular locking dependency detected [ 63.484091] 4.12.0-kfd-ozeng #3 Not tainted [ 63.488531] ------------------------------------------------------ [ 63.495146] HelloWorldLoop/2526 is trying to acquire lock: [ 63.501011] (&mm->mmap_sem){++++++}, at: [<ffffffff911898ce>] __might_fault+0x3e/0x90 [ 63.509472] but task is already holding lock: [ 63.515716] (&adev->srbm_mutex){+.+...}, at: [<ffffffffc0484feb>] lock_srbm+0x2b/0x50 [amdgpu] [ 63.525099] which lock already depends on the new lock. [ 63.533841] the existing dependency chain (in reverse order) is: [ 63.541839] -> #2 (&adev->srbm_mutex){+.+...}: [ 63.548178] lock_acquire+0x6d/0x90 [ 63.552461] __mutex_lock+0x70/0x8c0 [ 63.556826] mutex_lock_nested+0x16/0x20 [ 63.561603] gfx_v8_0_kiq_resume+0x1039/0x14a0 [amdgpu] [ 63.567817] gfx_v8_0_hw_init+0x204d/0x2210 [amdgpu] [ 63.573675] amdgpu_device_init+0xdea/0x1790 [amdgpu] [ 63.579640] amdgpu_driver_load_kms+0x63/0x220 [amdgpu] [ 63.585743] drm_dev_register+0x145/0x1e0 [ 63.590605] amdgpu_pci_probe+0x11e/0x160 [amdgpu] [ 63.596266] local_pci_probe+0x40/0xa0 [ 63.600803] pci_device_probe+0x134/0x150 [ 63.605650] driver_probe_device+0x2a1/0x460 [ 63.610785] __driver_attach+0xdc/0xe0 [ 63.615321] bus_for_each_dev+0x5f/0x90 [ 63.619984] driver_attach+0x19/0x20 [ 63.624337] bus_add_driver+0x40/0x270 [ 63.628908] driver_register+0x5b/0xe0 [ 63.633446] __pci_register_driver+0x5b/0x60 [ 63.638586] rtsx_pci_switch_output_voltage+0x1d/0x20 [rtsx_pci] [ 63.645564] do_one_initcall+0x4c/0x1b0 [ 63.650205] do_init_module+0x56/0x1ea [ 63.654767] load_module+0x208c/0x27d0 [ 63.659335] SYSC_finit_module+0x96/0xd0 [ 63.664058] SyS_finit_module+0x9/0x10 [ 63.668629] entry_SYSCALL_64_fastpath+0x1f/0xbe [ 63.674088] -> #1 (reservation_ww_class_mutex){+.+.+.}: [ 63.681257] lock_acquire+0x6d/0x90 [ 63.685551] __ww_mutex_lock.constprop.11+0x8c/0xed0 [ 63.691426] ww_mutex_lock+0x67/0x70 [ 63.695802] amdgpu_verify_access+0x6d/0x100 [amdgpu] [ 63.701743] ttm_bo_mmap+0x8e/0x100 [ttm] [ 63.706615] amdgpu_bo_mmap+0xd/0x60 [amdgpu] [ 63.711814] amdgpu_mmap+0x35/0x40 [amdgpu] [ 63.716904] mmap_region+0x3b5/0x5a0 [ 63.721255] do_mmap+0x400/0x4d0 [ 63.725260] vm_mmap_pgoff+0xb0/0xf0 [ 63.729625] SyS_mmap_pgoff+0x19e/0x260 [ 63.734292] SyS_mmap+0x1d/0x20 [ 63.738199] entry_SYSCALL_64_fastpath+0x1f/0xbe [ 63.743681] -> #0 (&mm->mmap_sem){++++++}: [ 63.749641] __lock_acquire+0x1401/0x1420 [ 63.754491] lock_acquire+0x6d/0x90 [ 63.758750] __might_fault+0x6b/0x90 [ 63.763176] kgd_hqd_load+0x24f/0x270 [amdgpu] [ 63.768432] load_mqd+0x4b/0x50 [amdkfd] [ 63.773192] create_queue_nocpsch+0x535/0x620 [amdkfd] [ 63.779237] pqm_create_queue+0x34d/0x4f0 [amdkfd] [ 63.784835] kfd_ioctl_create_queue+0x282/0x670 [amdkfd] [ 63.790973] kfd_ioctl+0x310/0x4d0 [amdkfd] [ 63.795944] do_vfs_ioctl+0x90/0x6e0 [ 63.800268] SyS_ioctl+0x74/0x80 [ 63.804207] entry_SYSCALL_64_fastpath+0x1f/0xbe [ 63.809607] other info that might help us debug this: [ 63.818026] Chain exists of: &mm->mmap_sem --> reservation_ww_class_mutex --> &adev->srbm_mutex [ 63.830382] Possible unsafe locking scenario: [ 63.836605] CPU0 CPU1 [ 63.841364] ---- ---- [ 63.846123] lock(&adev->srbm_mutex); [ 63.850061] lock(reservation_ww_class_mutex); [ 63.857475] lock(&adev->srbm_mutex); [ 63.864084] lock(&mm->mmap_sem); [ 63.867657] *** DEADLOCK *** [ 63.873884] 3 locks held by HelloWorldLoop/2526: [ 63.878739] #0: (&process->mutex){+.+.+.}, at: [<ffffffffc06e1a9a>] kfd_ioctl_create_queue+0x24a/0x670 [amdkfd] [ 63.889543] #1: (&dqm->lock){+.+...}, at: [<ffffffffc06eedeb>] create_queue_nocpsch+0x3b/0x620 [amdkfd] [ 63.899684] #2: (&adev->srbm_mutex){+.+...}, at: [<ffffffffc0484feb>] lock_srbm+0x2b/0x50 [amdgpu] [ 63.909500] stack backtrace: [ 63.914187] CPU: 3 PID: 2526 Comm: HelloWorldLoop Not tainted 4.12.0-kfd-ozeng #3 [ 63.922184] Hardware name: AMD Carrizo/Gardenia, BIOS WGA5819N_Weekly_15_08_1 08/19/2015 [ 63.930865] Call Trace: [ 63.933464] dump_stack+0x85/0xc9 [ 63.936999] print_circular_bug+0x1f9/0x207 [ 63.941442] __lock_acquire+0x1401/0x1420 [ 63.945745] ? lock_srbm+0x2b/0x50 [amdgpu] [ 63.950185] lock_acquire+0x6d/0x90 [ 63.953885] ? __might_fault+0x3e/0x90 [ 63.957899] __might_fault+0x6b/0x90 [ 63.961699] ? __might_fault+0x3e/0x90 [ 63.965755] kgd_hqd_load+0x24f/0x270 [amdgpu] [ 63.970577] load_mqd+0x4b/0x50 [amdkfd] [ 63.974745] create_queue_nocpsch+0x535/0x620 [amdkfd] [ 63.980242] pqm_create_queue+0x34d/0x4f0 [amdkfd] [ 63.985320] kfd_ioctl_create_queue+0x282/0x670 [amdkfd] [ 63.991021] kfd_ioctl+0x310/0x4d0 [amdkfd] [ 63.995499] ? kfd_ioctl_destroy_queue+0x70/0x70 [amdkfd] [ 64.001234] do_vfs_ioctl+0x90/0x6e0 [ 64.005065] ? up_read+0x1a/0x40 [ 64.008496] SyS_ioctl+0x74/0x80 [ 64.011955] entry_SYSCALL_64_fastpath+0x1f/0xbe [ 64.016863] RIP: 0033:0x7f4b3bd35f07 [ 64.020696] RSP: 002b:00007ffe7689ec38 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 64.028786] RAX: ffffffffffffffda RBX: 00000000002a2000 RCX: 00007f4b3bd35f07 [ 64.036414] RDX: 00007ffe7689ecb0 RSI: 00000000c0584b02 RDI: 0000000000000005 [ 64.044045] RBP: 00007f4a3212d000 R08: 00007f4b3c919000 R09: 0000000000080000 [ 64.051674] R10: 00007f4b376b64b8 R11: 0000000000000246 R12: 00007f4a3212d000 [ 64.059324] R13: 0000000000000015 R14: 0000000000000064 R15: 00007ffe7689ef50 Signed-off-by: Oak Zeng <Oak.Zeng@amd.com> Reviewed-by: Felix Kuehling <Felix.Kuehling@amd.com>

The idea to break the circular lock dependency is to temporarily drop dqm lock before calling allocate_mqd. See callstack #1 below. [ 59.510149] [drm] Initialized amdgpu 3.30.0 20150101 for 0000:04:00.0 on minor 0 [ 513.604034] ====================================================== [ 513.604205] WARNING: possible circular locking dependency detected [ 513.604375] 4.18.0-kfd-root #2 Tainted: G W [ 513.604530] ------------------------------------------------------ [ 513.604699] kswapd0/611 is trying to acquire lock: [ 513.604840] 00000000d254022e (&dqm->lock_hidden){+.+.}, at: evict_process_queues_nocpsch+0x26/0x140 [amdgpu] [ 513.605150] but task is already holding lock: [ 513.605307] 00000000961547fc (&anon_vma->rwsem){++++}, at: page_lock_anon_vma_read+0xe4/0x250 [ 513.605540] which lock already depends on the new lock. [ 513.605747] the existing dependency chain (in reverse order) is: [ 513.605944] -> #4 (&anon_vma->rwsem){++++}: [ 513.606106] __vma_adjust+0x147/0x7f0 [ 513.606231] __split_vma+0x179/0x190 [ 513.606353] mprotect_fixup+0x217/0x260 [ 513.606553] do_mprotect_pkey+0x211/0x380 [ 513.606752] __x64_sys_mprotect+0x1b/0x20 [ 513.606954] do_syscall_64+0x50/0x1a0 [ 513.607149] entry_SYSCALL_64_after_hwframe+0x49/0xbe [ 513.607380] -> #3 (&mapping->i_mmap_rwsem){++++}: [ 513.607678] rmap_walk_file+0x1f0/0x280 [ 513.607887] page_referenced+0xdd/0x180 [ 513.608081] shrink_page_list+0x853/0xcb0 [ 513.608279] shrink_inactive_list+0x33b/0x700 [ 513.608483] shrink_node_memcg+0x37a/0x7f0 [ 513.608682] shrink_node+0xd8/0x490 [ 513.608869] balance_pgdat+0x18b/0x3b0 [ 513.609062] kswapd+0x203/0x5c0 [ 513.609241] kthread+0x100/0x140 [ 513.609420] ret_from_fork+0x24/0x30 [ 513.609607] -> #2 (fs_reclaim){+.+.}: [ 513.609883] kmem_cache_alloc_trace+0x34/0x2e0 [ 513.610093] reservation_object_reserve_shared+0x139/0x300 [ 513.610326] ttm_bo_init_reserved+0x291/0x480 [ttm] [ 513.610567] amdgpu_bo_do_create+0x1d2/0x650 [amdgpu] [ 513.610811] amdgpu_bo_create+0x40/0x1f0 [amdgpu] [ 513.611041] amdgpu_bo_create_reserved+0x249/0x2d0 [amdgpu] [ 513.611290] amdgpu_bo_create_kernel+0x12/0x70 [amdgpu] [ 513.611584] amdgpu_ttm_init+0x2cb/0x560 [amdgpu] [ 513.611823] gmc_v9_0_sw_init+0x400/0x750 [amdgpu] [ 513.612491] amdgpu_device_init+0x14eb/0x1990 [amdgpu] [ 513.612730] amdgpu_driver_load_kms+0x78/0x290 [amdgpu] [ 513.612958] drm_dev_register+0x111/0x1a0 [ 513.613171] amdgpu_pci_probe+0x11c/0x1e0 [amdgpu] [ 513.613389] local_pci_probe+0x3f/0x90 [ 513.613581] pci_device_probe+0x102/0x1c0 [ 513.613779] driver_probe_device+0x2a7/0x480 [ 513.613984] __driver_attach+0x10a/0x110 [ 513.614179] bus_for_each_dev+0x67/0xc0 [ 513.614372] bus_add_driver+0x1eb/0x260 [ 513.614565] driver_register+0x5b/0xe0 [ 513.614756] do_one_initcall+0xac/0x357 [ 513.614952] do_init_module+0x5b/0x213 [ 513.615145] load_module+0x2542/0x2d30 [ 513.615337] __do_sys_finit_module+0xd2/0x100 [ 513.615541] do_syscall_64+0x50/0x1a0 [ 513.615731] entry_SYSCALL_64_after_hwframe+0x49/0xbe [ 513.615963] -> #1 (reservation_ww_class_mutex){+.+.}: [ 513.616293] amdgpu_amdkfd_alloc_gtt_mem+0xcf/0x2c0 [amdgpu] [ 513.616554] init_mqd+0x223/0x260 [amdgpu] [ 513.616779] create_queue_nocpsch+0x4d9/0x600 [amdgpu] [ 513.617031] pqm_create_queue+0x37c/0x520 [amdgpu] [ 513.617270] kfd_ioctl_create_queue+0x2f9/0x650 [amdgpu] [ 513.617522] kfd_ioctl+0x202/0x350 [amdgpu] [ 513.617724] do_vfs_ioctl+0x9f/0x6c0 [ 513.617914] ksys_ioctl+0x66/0x70 [ 513.618095] __x64_sys_ioctl+0x16/0x20 [ 513.618286] do_syscall_64+0x50/0x1a0 [ 513.618476] entry_SYSCALL_64_after_hwframe+0x49/0xbe [ 513.618695] -> #0 (&dqm->lock_hidden){+.+.}: [ 513.618984] __mutex_lock+0x98/0x970 [ 513.619197] evict_process_queues_nocpsch+0x26/0x140 [amdgpu] [ 513.619459] kfd_process_evict_queues+0x3b/0xb0 [amdgpu] [ 513.619710] kgd2kfd_quiesce_mm+0x1c/0x40 [amdgpu] [ 513.620103] amdgpu_amdkfd_evict_userptr+0x38/0x70 [amdgpu] [ 513.620363] amdgpu_mn_invalidate_range_start_hsa+0xa6/0xc0 [amdgpu] [ 513.620614] __mmu_notifier_invalidate_range_start+0x70/0xb0 [ 513.620851] try_to_unmap_one+0x7fc/0x8f0 [ 513.621049] rmap_walk_anon+0x121/0x290 [ 513.621242] try_to_unmap+0x93/0xf0 [ 513.621428] shrink_page_list+0x606/0xcb0 [ 513.621625] shrink_inactive_list+0x33b/0x700 [ 513.621835] shrink_node_memcg+0x37a/0x7f0 [ 513.622034] shrink_node+0xd8/0x490 [ 513.622219] balance_pgdat+0x18b/0x3b0 [ 513.622410] kswapd+0x203/0x5c0 [ 513.622589] kthread+0x100/0x140 [ 513.622769] ret_from_fork+0x24/0x30 [ 513.622957] other info that might help us debug this: [ 513.623354] Chain exists of: &dqm->lock_hidden --> &mapping->i_mmap_rwsem --> &anon_vma->rwsem [ 513.623900] Possible unsafe locking scenario: [ 513.624189] CPU0 CPU1 [ 513.624397] ---- ---- [ 513.624594] lock(&anon_vma->rwsem); [ 513.624771] lock(&mapping->i_mmap_rwsem); [ 513.625020] lock(&anon_vma->rwsem); [ 513.625253] lock(&dqm->lock_hidden); [ 513.625433] *** DEADLOCK *** [ 513.625783] 3 locks held by kswapd0/611: [ 513.625967] #0: 00000000f14edf84 (fs_reclaim){+.+.}, at: __fs_reclaim_acquire+0x5/0x30 [ 513.626309] #1: 00000000961547fc (&anon_vma->rwsem){++++}, at: page_lock_anon_vma_read+0xe4/0x250 [ 513.626671] #2: 0000000067b5cd12 (srcu){....}, at: __mmu_notifier_invalidate_range_start+0x5/0xb0 [ 513.627037] stack backtrace: [ 513.627292] CPU: 0 PID: 611 Comm: kswapd0 Tainted: G W 4.18.0-kfd-root #2 [ 513.627632] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 [ 513.627990] Call Trace: [ 513.628143] dump_stack+0x7c/0xbb [ 513.628315] print_circular_bug.isra.37+0x21b/0x228 [ 513.628581] __lock_acquire+0xf7d/0x1470 [ 513.628782] ? unwind_next_frame+0x6c/0x4f0 [ 513.628974] ? lock_acquire+0xec/0x1e0 [ 513.629154] lock_acquire+0xec/0x1e0 [ 513.629357] ? evict_process_queues_nocpsch+0x26/0x140 [amdgpu] [ 513.629587] __mutex_lock+0x98/0x970 [ 513.629790] ? evict_process_queues_nocpsch+0x26/0x140 [amdgpu] [ 513.630047] ? evict_process_queues_nocpsch+0x26/0x140 [amdgpu] [ 513.630309] ? evict_process_queues_nocpsch+0x26/0x140 [amdgpu] [ 513.630562] evict_process_queues_nocpsch+0x26/0x140 [amdgpu] [ 513.630816] kfd_process_evict_queues+0x3b/0xb0 [amdgpu] [ 513.631057] kgd2kfd_quiesce_mm+0x1c/0x40 [amdgpu] [ 513.631288] amdgpu_amdkfd_evict_userptr+0x38/0x70 [amdgpu] [ 513.631536] amdgpu_mn_invalidate_range_start_hsa+0xa6/0xc0 [amdgpu] [ 513.632076] __mmu_notifier_invalidate_range_start+0x70/0xb0 [ 513.632299] try_to_unmap_one+0x7fc/0x8f0 [ 513.632487] ? page_lock_anon_vma_read+0x68/0x250 [ 513.632690] rmap_walk_anon+0x121/0x290 [ 513.632875] try_to_unmap+0x93/0xf0 [ 513.633050] ? page_remove_rmap+0x330/0x330 [ 513.633239] ? rcu_read_unlock+0x60/0x60 [ 513.633422] ? page_get_anon_vma+0x160/0x160 [ 513.633613] shrink_page_list+0x606/0xcb0 [ 513.633800] shrink_inactive_list+0x33b/0x700 [ 513.633997] shrink_node_memcg+0x37a/0x7f0 [ 513.634186] ? shrink_node+0xd8/0x490 [ 513.634363] shrink_node+0xd8/0x490 [ 513.634537] balance_pgdat+0x18b/0x3b0 [ 513.634718] kswapd+0x203/0x5c0 [ 513.634887] ? wait_woken+0xb0/0xb0 [ 513.635062] kthread+0x100/0x140 [ 513.635231] ? balance_pgdat+0x3b0/0x3b0 [ 513.635414] ? kthread_delayed_work_timer_fn+0x80/0x80 [ 513.635626] ret_from_fork+0x24/0x30 [ 513.636042] Evicting PASID 32768 queues [ 513.936236] Restoring PASID 32768 queues [ 524.708912] Evicting PASID 32768 queues [ 524.999875] Restoring PASID 32768 queues Change-Id: I8827178d3d884d793a140a0bb4ec1d89f5dc50f6 Signed-off-by: Oak Zeng <Oak.Zeng@amd.com> Reviewed-by: Philip Yang <philip.yang@amd.com>

The idea to break the circular lock dependency is to temporarily drop dqm lock before calling allocate_mqd. See callstack #1 below. [ 59.510149] [drm] Initialized amdgpu 3.30.0 20150101 for 0000:04:00.0 on minor 0 [ 513.604034] ====================================================== [ 513.604205] WARNING: possible circular locking dependency detected [ 513.604375] 4.18.0-kfd-root #2 Tainted: G W [ 513.604530] ------------------------------------------------------ [ 513.604699] kswapd0/611 is trying to acquire lock: [ 513.604840] 00000000d254022e (&dqm->lock_hidden){+.+.}, at: evict_process_queues_nocpsch+0x26/0x140 [amdgpu] [ 513.605150] but task is already holding lock: [ 513.605307] 00000000961547fc (&anon_vma->rwsem){++++}, at: page_lock_anon_vma_read+0xe4/0x250 [ 513.605540] which lock already depends on the new lock. [ 513.605747] the existing dependency chain (in reverse order) is: [ 513.605944] -> #4 (&anon_vma->rwsem){++++}: [ 513.606106] __vma_adjust+0x147/0x7f0 [ 513.606231] __split_vma+0x179/0x190 [ 513.606353] mprotect_fixup+0x217/0x260 [ 513.606553] do_mprotect_pkey+0x211/0x380 [ 513.606752] __x64_sys_mprotect+0x1b/0x20 [ 513.606954] do_syscall_64+0x50/0x1a0 [ 513.607149] entry_SYSCALL_64_after_hwframe+0x49/0xbe [ 513.607380] -> #3 (&mapping->i_mmap_rwsem){++++}: [ 513.607678] rmap_walk_file+0x1f0/0x280 [ 513.607887] page_referenced+0xdd/0x180 [ 513.608081] shrink_page_list+0x853/0xcb0 [ 513.608279] shrink_inactive_list+0x33b/0x700 [ 513.608483] shrink_node_memcg+0x37a/0x7f0 [ 513.608682] shrink_node+0xd8/0x490 [ 513.608869] balance_pgdat+0x18b/0x3b0 [ 513.609062] kswapd+0x203/0x5c0 [ 513.609241] kthread+0x100/0x140 [ 513.609420] ret_from_fork+0x24/0x30 [ 513.609607] -> #2 (fs_reclaim){+.+.}: [ 513.609883] kmem_cache_alloc_trace+0x34/0x2e0 [ 513.610093] reservation_object_reserve_shared+0x139/0x300 [ 513.610326] ttm_bo_init_reserved+0x291/0x480 [ttm] [ 513.610567] amdgpu_bo_do_create+0x1d2/0x650 [amdgpu] [ 513.610811] amdgpu_bo_create+0x40/0x1f0 [amdgpu] [ 513.611041] amdgpu_bo_create_reserved+0x249/0x2d0 [amdgpu] [ 513.611290] amdgpu_bo_create_kernel+0x12/0x70 [amdgpu] [ 513.611584] amdgpu_ttm_init+0x2cb/0x560 [amdgpu] [ 513.611823] gmc_v9_0_sw_init+0x400/0x750 [amdgpu] [ 513.612491] amdgpu_device_init+0x14eb/0x1990 [amdgpu] [ 513.612730] amdgpu_driver_load_kms+0x78/0x290 [amdgpu] [ 513.612958] drm_dev_register+0x111/0x1a0 [ 513.613171] amdgpu_pci_probe+0x11c/0x1e0 [amdgpu] [ 513.613389] local_pci_probe+0x3f/0x90 [ 513.613581] pci_device_probe+0x102/0x1c0 [ 513.613779] driver_probe_device+0x2a7/0x480 [ 513.613984] __driver_attach+0x10a/0x110 [ 513.614179] bus_for_each_dev+0x67/0xc0 [ 513.614372] bus_add_driver+0x1eb/0x260 [ 513.614565] driver_register+0x5b/0xe0 [ 513.614756] do_one_initcall+0xac/0x357 [ 513.614952] do_init_module+0x5b/0x213 [ 513.615145] load_module+0x2542/0x2d30 [ 513.615337] __do_sys_finit_module+0xd2/0x100 [ 513.615541] do_syscall_64+0x50/0x1a0 [ 513.615731] entry_SYSCALL_64_after_hwframe+0x49/0xbe [ 513.615963] -> #1 (reservation_ww_class_mutex){+.+.}: [ 513.616293] amdgpu_amdkfd_alloc_gtt_mem+0xcf/0x2c0 [amdgpu] [ 513.616554] init_mqd+0x223/0x260 [amdgpu] [ 513.616779] create_queue_nocpsch+0x4d9/0x600 [amdgpu] [ 513.617031] pqm_create_queue+0x37c/0x520 [amdgpu] [ 513.617270] kfd_ioctl_create_queue+0x2f9/0x650 [amdgpu] [ 513.617522] kfd_ioctl+0x202/0x350 [amdgpu] [ 513.617724] do_vfs_ioctl+0x9f/0x6c0 [ 513.617914] ksys_ioctl+0x66/0x70 [ 513.618095] __x64_sys_ioctl+0x16/0x20 [ 513.618286] do_syscall_64+0x50/0x1a0 [ 513.618476] entry_SYSCALL_64_after_hwframe+0x49/0xbe [ 513.618695] -> #0 (&dqm->lock_hidden){+.+.}: [ 513.618984] __mutex_lock+0x98/0x970 [ 513.619197] evict_process_queues_nocpsch+0x26/0x140 [amdgpu] [ 513.619459] kfd_process_evict_queues+0x3b/0xb0 [amdgpu] [ 513.619710] kgd2kfd_quiesce_mm+0x1c/0x40 [amdgpu] [ 513.620103] amdgpu_amdkfd_evict_userptr+0x38/0x70 [amdgpu] [ 513.620363] amdgpu_mn_invalidate_range_start_hsa+0xa6/0xc0 [amdgpu] [ 513.620614] __mmu_notifier_invalidate_range_start+0x70/0xb0 [ 513.620851] try_to_unmap_one+0x7fc/0x8f0 [ 513.621049] rmap_walk_anon+0x121/0x290 [ 513.621242] try_to_unmap+0x93/0xf0 [ 513.621428] shrink_page_list+0x606/0xcb0 [ 513.621625] shrink_inactive_list+0x33b/0x700 [ 513.621835] shrink_node_memcg+0x37a/0x7f0 [ 513.622034] shrink_node+0xd8/0x490 [ 513.622219] balance_pgdat+0x18b/0x3b0 [ 513.622410] kswapd+0x203/0x5c0 [ 513.622589] kthread+0x100/0x140 [ 513.622769] ret_from_fork+0x24/0x30 [ 513.622957] other info that might help us debug this: [ 513.623354] Chain exists of: &dqm->lock_hidden --> &mapping->i_mmap_rwsem --> &anon_vma->rwsem [ 513.623900] Possible unsafe locking scenario: [ 513.624189] CPU0 CPU1 [ 513.624397] ---- ---- [ 513.624594] lock(&anon_vma->rwsem); [ 513.624771] lock(&mapping->i_mmap_rwsem); [ 513.625020] lock(&anon_vma->rwsem); [ 513.625253] lock(&dqm->lock_hidden); [ 513.625433] *** DEADLOCK *** [ 513.625783] 3 locks held by kswapd0/611: [ 513.625967] #0: 00000000f14edf84 (fs_reclaim){+.+.}, at: __fs_reclaim_acquire+0x5/0x30 [ 513.626309] #1: 00000000961547fc (&anon_vma->rwsem){++++}, at: page_lock_anon_vma_read+0xe4/0x250 [ 513.626671] #2: 0000000067b5cd12 (srcu){....}, at: __mmu_notifier_invalidate_range_start+0x5/0xb0 [ 513.627037] stack backtrace: [ 513.627292] CPU: 0 PID: 611 Comm: kswapd0 Tainted: G W 4.18.0-kfd-root #2 [ 513.627632] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 [ 513.627990] Call Trace: [ 513.628143] dump_stack+0x7c/0xbb [ 513.628315] print_circular_bug.isra.37+0x21b/0x228 [ 513.628581] __lock_acquire+0xf7d/0x1470 [ 513.628782] ? unwind_next_frame+0x6c/0x4f0 [ 513.628974] ? lock_acquire+0xec/0x1e0 [ 513.629154] lock_acquire+0xec/0x1e0 [ 513.629357] ? evict_process_queues_nocpsch+0x26/0x140 [amdgpu] [ 513.629587] __mutex_lock+0x98/0x970 [ 513.629790] ? evict_process_queues_nocpsch+0x26/0x140 [amdgpu] [ 513.630047] ? evict_process_queues_nocpsch+0x26/0x140 [amdgpu] [ 513.630309] ? evict_process_queues_nocpsch+0x26/0x140 [amdgpu] [ 513.630562] evict_process_queues_nocpsch+0x26/0x140 [amdgpu] [ 513.630816] kfd_process_evict_queues+0x3b/0xb0 [amdgpu] [ 513.631057] kgd2kfd_quiesce_mm+0x1c/0x40 [amdgpu] [ 513.631288] amdgpu_amdkfd_evict_userptr+0x38/0x70 [amdgpu] [ 513.631536] amdgpu_mn_invalidate_range_start_hsa+0xa6/0xc0 [amdgpu] [ 513.632076] __mmu_notifier_invalidate_range_start+0x70/0xb0 [ 513.632299] try_to_unmap_one+0x7fc/0x8f0 [ 513.632487] ? page_lock_anon_vma_read+0x68/0x250 [ 513.632690] rmap_walk_anon+0x121/0x290 [ 513.632875] try_to_unmap+0x93/0xf0 [ 513.633050] ? page_remove_rmap+0x330/0x330 [ 513.633239] ? rcu_read_unlock+0x60/0x60 [ 513.633422] ? page_get_anon_vma+0x160/0x160 [ 513.633613] shrink_page_list+0x606/0xcb0 [ 513.633800] shrink_inactive_list+0x33b/0x700 [ 513.633997] shrink_node_memcg+0x37a/0x7f0 [ 513.634186] ? shrink_node+0xd8/0x490 [ 513.634363] shrink_node+0xd8/0x490 [ 513.634537] balance_pgdat+0x18b/0x3b0 [ 513.634718] kswapd+0x203/0x5c0 [ 513.634887] ? wait_woken+0xb0/0xb0 [ 513.635062] kthread+0x100/0x140 [ 513.635231] ? balance_pgdat+0x3b0/0x3b0 [ 513.635414] ? kthread_delayed_work_timer_fn+0x80/0x80 [ 513.635626] ret_from_fork+0x24/0x30 [ 513.636042] Evicting PASID 32768 queues [ 513.936236] Restoring PASID 32768 queues [ 524.708912] Evicting PASID 32768 queues [ 524.999875] Restoring PASID 32768 queues Change-Id: I89b4432cdc6210fcddc5204f5a90b3d4df4b883c Signed-off-by: Oak Zeng <Oak.Zeng@amd.com> Reviewed-by: Philip Yang <philip.yang@amd.com>

[Why] DRM private objects have no hw_done/flip_done fencing mechanism on their own and cannot be used to sequence commits accordingly. When issuing commits that don't touch the same set of hardware resources like page-flips on different CRTCs we can run into the issue below because of this: 1. Client requests non-blocking Commit #1, has a new dc_state #1, state is swapped, commit tail is deferred to work queue 2. Client requests non-blocking Commit #2, has a new dc_state #2, state is swapped, commit tail is deferred to work queue 3. Commit #2 work starts, commit tail finishes, atomic state is cleared, dc_state #1 is freed 4. Commit #1 work starts, commit tail encounters null pointer deref on dc_state #1 In order to change the DC state as in the private object we need to ensure that we wait for all outstanding commits to finish and that any other pending commits must wait for the current one to finish as well. We do this for MEDIUM and FULL updates. But not for FAST updates, nor would we want to since it would cause stuttering from the delays. FAST updates that go through dm_determine_update_type_for_commit always create a new dc_state and lock the DRM private object if there are any changed planes. We need the old state to validate, but we don't actually need the new state here. [How] If the commit isn't a full update then the use after free can be resolved by simply discarding the new state entirely and retaining the existing one instead. With this change the sequence above can be reexamined. Commit #2 will still free Commit #1's reference, but before this happens we actually added an additional reference as part of Commit #2. If an update comes in during this that needs to change the dc_state it will need to wait on Commit #1 and Commit #2 to finish. Then it'll swap the state, finish the work in commit tail and drop the last reference on Commit #2's dc_state. Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=204181 Fixes: 004b393 ("drm/amd/display: Check scaling info when determing update type") Signed-off-by: Nicholas Kazlauskas <nicholas.kazlauskas@amd.com> Acked-by: Alex Deucher <alexander.deucher@amd.com> Reviewed-by: David Francis <david.francis@amd.com>

[ 150.887733] ====================================================== [ 150.893903] WARNING: possible circular locking dependency detected [ 150.905917] ------------------------------------------------------ [ 150.912129] kfdtest/4081 is trying to acquire lock: [ 150.917002] ffff8f7f3762e118 (&mm->mmap_sem#2){++++}, at: __might_fault+0x3e/0x90 [ 150.924490] but task is already holding lock: [ 150.930320] ffff8f7f49d229e8 (&dqm->lock_hidden){+.+.}, at: destroy_queue_cpsch+0x29/0x210 [amdgpu] [ 150.939432] which lock already depends on the new lock. [ 150.947603] the existing dependency chain (in reverse order) is: [ 150.955074] -> #3 (&dqm->lock_hidden){+.+.}: [ 150.960822] __mutex_lock+0xa1/0x9f0 [ 150.964996] evict_process_queues_cpsch+0x22/0x120 [amdgpu] [ 150.971155] kfd_process_evict_queues+0x3b/0xc0 [amdgpu] [ 150.977054] kgd2kfd_quiesce_mm+0x25/0x60 [amdgpu] [ 150.982442] amdgpu_amdkfd_evict_userptr+0x35/0x70 [amdgpu] [ 150.988615] amdgpu_mn_invalidate_hsa+0x41/0x60 [amdgpu] [ 150.994448] __mmu_notifier_invalidate_range_start+0xa4/0x240 [ 151.000714] copy_page_range+0xd70/0xd80 [ 151.005159] dup_mm+0x3ca/0x550 [ 151.008816] copy_process+0x1bdc/0x1c70 [ 151.013183] _do_fork+0x76/0x6c0 [ 151.016929] __x64_sys_clone+0x8c/0xb0 [ 151.021201] do_syscall_64+0x4a/0x1d0 [ 151.025404] entry_SYSCALL_64_after_hwframe+0x49/0xbe [ 151.030977] -> #2 (&adev->notifier_lock){+.+.}: [ 151.036993] __mutex_lock+0xa1/0x9f0 [ 151.041168] amdgpu_mn_invalidate_hsa+0x30/0x60 [amdgpu] [ 151.047019] __mmu_notifier_invalidate_range_start+0xa4/0x240 [ 151.053277] copy_page_range+0xd70/0xd80 [ 151.057722] dup_mm+0x3ca/0x550 [ 151.061388] copy_process+0x1bdc/0x1c70 [ 151.065748] _do_fork+0x76/0x6c0 [ 151.069499] __x64_sys_clone+0x8c/0xb0 [ 151.073765] do_syscall_64+0x4a/0x1d0 [ 151.077952] entry_SYSCALL_64_after_hwframe+0x49/0xbe [ 151.083523] -> #1 (mmu_notifier_invalidate_range_start){+.+.}: [ 151.090833] change_protection+0x802/0xab0 [ 151.095448] mprotect_fixup+0x187/0x2d0 [ 151.099801] setup_arg_pages+0x124/0x250 [ 151.104251] load_elf_binary+0x3a4/0x1464 [ 151.108781] search_binary_handler+0x6c/0x210 [ 151.113656] __do_execve_file.isra.40+0x7f7/0xa50 [ 151.118875] do_execve+0x21/0x30 [ 151.122632] call_usermodehelper_exec_async+0x17e/0x190 [ 151.128393] ret_from_fork+0x24/0x30 [ 151.132489] -> #0 (&mm->mmap_sem#2){++++}: [ 151.138064] __lock_acquire+0x11a1/0x1490 [ 151.142597] lock_acquire+0x90/0x180 [ 151.146694] __might_fault+0x68/0x90 [ 151.150879] read_sdma_queue_counter+0x5f/0xb0 [amdgpu] [ 151.156693] update_sdma_queue_past_activity_stats+0x3b/0x90 [amdgpu] [ 151.163725] destroy_queue_cpsch+0x1ae/0x210 [amdgpu] [ 151.169373] pqm_destroy_queue+0xf0/0x250 [amdgpu] [ 151.174762] kfd_ioctl_destroy_queue+0x32/0x70 [amdgpu] [ 151.180577] kfd_ioctl+0x223/0x400 [amdgpu] [ 151.185284] ksys_ioctl+0x8f/0xb0 [ 151.189118] __x64_sys_ioctl+0x16/0x20 [ 151.193389] do_syscall_64+0x4a/0x1d0 [ 151.197569] entry_SYSCALL_64_after_hwframe+0x49/0xbe [ 151.203141] other info that might help us debug this: [ 151.211140] Chain exists of: &mm->mmap_sem#2 --> &adev->notifier_lock --> &dqm->lock_hidden [ 151.222535] Possible unsafe locking scenario: [ 151.228447] CPU0 CPU1 [ 151.232971] ---- ---- [ 151.237502] lock(&dqm->lock_hidden); [ 151.241254] lock(&adev->notifier_lock); [ 151.247774] lock(&dqm->lock_hidden); [ 151.254038] lock(&mm->mmap_sem#2); This commit fixes the warning by ensuring get_user() is not called while reading SDMA stats with dqm_lock held as get_user() could cause a page fault which leads to the circular locking scenario. Signed-off-by: Mukul Joshi <mukul.joshi@amd.com> Reviewed-by: Felix Kuehling <Felix.Kuehling@amd.com>

This patch fixes a race condition that causes a use-after-free during amdgpu_dm_atomic_commit_tail. This can occur when 2 non-blocking commits are requested and the second one finishes before the first. Essentially, this bug occurs when the following sequence of events happens: 1. Non-blocking commit #1 is requested w/ a new dm_state #1 and is deferred to the workqueue. 2. Non-blocking commit #2 is requested w/ a new dm_state #2 and is deferred to the workqueue. 3. Commit #2 starts before commit #1, dm_state #1 is used in the commit_tail and commit #2 completes, freeing dm_state #1. 4. Commit #1 starts after commit #2 completes, uses the freed dm_state 1 and dereferences a freelist pointer while setting the context. Since this bug has only been spotted with fast commits, this patch fixes the bug by clearing the dm_state instead of using the old dc_state for fast updates. In addition, since dm_state is only used for its dc_state and amdgpu_dm_atomic_commit_tail will retain the dc_state if none is found, removing the dm_state should not have any consequences in fast updates. This use-after-free bug has existed for a while now, but only caused a noticeable issue starting from 5.7-rc1 due to 3202fa6 ("slub: relocate freelist pointer to middle of object") moving the freelist pointer from dm_state->base (which was unused) to dm_state->context (which is dereferenced). Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=207383 Fixes: bd200d1 ("drm/amd/display: Don't replace the dc_state for fast updates") Reported-by: Duncan <1i5t5.duncan@cox.net> Signed-off-by: Mazin Rezk <mnrzk@protonmail.com> Reviewed-by: Nicholas Kazlauskas <nicholas.kazlauskas@amd.com>

[ 584.110304] ============================================ [ 584.110590] WARNING: possible recursive locking detected [ 584.110876] 5.6.0-deli-v5.6-2848-g3f3109b0e75f #1 Tainted: G OE [ 584.111164] -------------------------------------------- [ 584.111456] kworker/38:1/553 is trying to acquire lock: [ 584.111721] ffff9b15ff0a47a0 (&adev->reset_sem){++++}, at: amdgpu_device_gpu_recover+0x262/0x1030 [amdgpu] [ 584.112112] but task is already holding lock: [ 584.112673] ffff9b1603d247a0 (&adev->reset_sem){++++}, at: amdgpu_device_gpu_recover+0x262/0x1030 [amdgpu] [ 584.113068] other info that might help us debug this: [ 584.113689] Possible unsafe locking scenario: [ 584.114350] CPU0 [ 584.114685] ---- [ 584.115014] lock(&adev->reset_sem); [ 584.115349] lock(&adev->reset_sem); [ 584.115678] *** DEADLOCK *** [ 584.116624] May be due to missing lock nesting notation [ 584.117284] 4 locks held by kworker/38:1/553: [ 584.117616] #0: ffff9ad635c1d348 ((wq_completion)events){+.+.}, at: process_one_work+0x21f/0x630 [ 584.117967] #1: ffffac708e1c3e58 ((work_completion)(&con->recovery_work)){+.+.}, at: process_one_work+0x21f/0x630 [ 584.118358] #2: ffffffffc1c2a5d0 (&tmp->hive_lock){+.+.}, at: amdgpu_device_gpu_recover+0xae/0x1030 [amdgpu] [ 584.118786] #3: ffff9b1603d247a0 (&adev->reset_sem){++++}, at: amdgpu_device_gpu_recover+0x262/0x1030 [amdgpu] [ 584.119222] stack backtrace: [ 584.119990] CPU: 38 PID: 553 Comm: kworker/38:1 Kdump: loaded Tainted: G OE 5.6.0-deli-v5.6-2848-g3f3109b0e75f #1 [ 584.120782] Hardware name: Supermicro SYS-7049GP-TRT/X11DPG-QT, BIOS 3.1 05/23/2019 [ 584.121223] Workqueue: events amdgpu_ras_do_recovery [amdgpu] [ 584.121638] Call Trace: [ 584.122050] dump_stack+0x98/0xd5 [ 584.122499] __lock_acquire+0x1139/0x16e0 [ 584.122931] ? trace_hardirqs_on+0x3b/0xf0 [ 584.123358] ? cancel_delayed_work+0xa6/0xc0 [ 584.123771] lock_acquire+0xb8/0x1c0 [ 584.124197] ? amdgpu_device_gpu_recover+0x262/0x1030 [amdgpu] [ 584.124599] down_write+0x49/0x120 [ 584.125032] ? amdgpu_device_gpu_recover+0x262/0x1030 [amdgpu] [ 584.125472] amdgpu_device_gpu_recover+0x262/0x1030 [amdgpu] [ 584.125910] ? amdgpu_ras_error_query+0x1b8/0x2a0 [amdgpu] [ 584.126367] amdgpu_ras_do_recovery+0x159/0x190 [amdgpu] [ 584.126789] process_one_work+0x29e/0x630 [ 584.127208] worker_thread+0x3c/0x3f0 [ 584.127621] ? __kthread_parkme+0x61/0x90 [ 584.128014] kthread+0x12f/0x150 [ 584.128402] ? process_one_work+0x630/0x630 [ 584.128790] ? kthread_park+0x90/0x90 [ 584.129174] ret_from_fork+0x3a/0x50 Each adev has owned lock_class_key to avoid false positive recursive locking. v2: 1. register adev->lock_key into lockdep, otherwise lockdep will report the below warning [ 1216.705820] BUG: key ffff890183b647d0 has not been registered! [ 1216.705924] ------------[ cut here ]------------ [ 1216.705972] DEBUG_LOCKS_WARN_ON(1) [ 1216.705997] WARNING: CPU: 20 PID: 541 at kernel/locking/lockdep.c:3743 lockdep_init_map+0x150/0x210 v3: change to use down_write_nest_lock to annotate the false dead-lock warning. Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch> Signed-off-by: Dennis Li <Dennis.Li@amd.com>

Re-apply commit 4fe159da08343cf2340593894d14cab896cf4c43 [ 584.110304] ============================================ [ 584.110590] WARNING: possible recursive locking detected [ 584.110876] 5.6.0-deli-v5.6-2848-g3f3109b0e75f #1 Tainted: G OE [ 584.111164] -------------------------------------------- [ 584.111456] kworker/38:1/553 is trying to acquire lock: [ 584.111721] ffff9b15ff0a47a0 (&adev->reset_sem){++++}, at: amdgpu_device_gpu_recover+0x262/0x1030 [amdgpu] [ 584.112112] but task is already holding lock: [ 584.112673] ffff9b1603d247a0 (&adev->reset_sem){++++}, at: amdgpu_device_gpu_recover+0x262/0x1030 [amdgpu] [ 584.113068] other info that might help us debug this: [ 584.113689] Possible unsafe locking scenario: [ 584.114350] CPU0 [ 584.114685] ---- [ 584.115014] lock(&adev->reset_sem); [ 584.115349] lock(&adev->reset_sem); [ 584.115678] *** DEADLOCK *** [ 584.116624] May be due to missing lock nesting notation [ 584.117284] 4 locks held by kworker/38:1/553: [ 584.117616] #0: ffff9ad635c1d348 ((wq_completion)events){+.+.}, at: process_one_work+0x21f/0x630 [ 584.117967] #1: ffffac708e1c3e58 ((work_completion)(&con->recovery_work)){+.+.}, at: process_one_work+0x21f/0x630 [ 584.118358] #2: ffffffffc1c2a5d0 (&tmp->hive_lock){+.+.}, at: amdgpu_device_gpu_recover+0xae/0x1030 [amdgpu] [ 584.118786] #3: ffff9b1603d247a0 (&adev->reset_sem){++++}, at: amdgpu_device_gpu_recover+0x262/0x1030 [amdgpu] [ 584.119222] stack backtrace: [ 584.119990] CPU: 38 PID: 553 Comm: kworker/38:1 Kdump: loaded Tainted: G OE 5.6.0-deli-v5.6-2848-g3f3109b0e75f #1 [ 584.120782] Hardware name: Supermicro SYS-7049GP-TRT/X11DPG-QT, BIOS 3.1 05/23/2019 [ 584.121223] Workqueue: events amdgpu_ras_do_recovery [amdgpu] [ 584.121638] Call Trace: [ 584.122050] dump_stack+0x98/0xd5 [ 584.122499] __lock_acquire+0x1139/0x16e0 [ 584.122931] ? trace_hardirqs_on+0x3b/0xf0 [ 584.123358] ? cancel_delayed_work+0xa6/0xc0 [ 584.123771] lock_acquire+0xb8/0x1c0 [ 584.124197] ? amdgpu_device_gpu_recover+0x262/0x1030 [amdgpu] [ 584.124599] down_write+0x49/0x120 [ 584.125032] ? amdgpu_device_gpu_recover+0x262/0x1030 [amdgpu] [ 584.125472] amdgpu_device_gpu_recover+0x262/0x1030 [amdgpu] [ 584.125910] ? amdgpu_ras_error_query+0x1b8/0x2a0 [amdgpu] [ 584.126367] amdgpu_ras_do_recovery+0x159/0x190 [amdgpu] [ 584.126789] process_one_work+0x29e/0x630 [ 584.127208] worker_thread+0x3c/0x3f0 [ 584.127621] ? __kthread_parkme+0x61/0x90 [ 584.128014] kthread+0x12f/0x150 [ 584.128402] ? process_one_work+0x630/0x630 [ 584.128790] ? kthread_park+0x90/0x90 [ 584.129174] ret_from_fork+0x3a/0x50 Each adev has owned lock_class_key to avoid false positive recursive locking. v2: 1. register adev->lock_key into lockdep, otherwise lockdep will report the below warning [ 1216.705820] BUG: key ffff890183b647d0 has not been registered! [ 1216.705924] ------------[ cut here ]------------ [ 1216.705972] DEBUG_LOCKS_WARN_ON(1) [ 1216.705997] WARNING: CPU: 20 PID: 541 at kernel/locking/lockdep.c:3743 lockdep_init_map+0x150/0x210 v3: change to use down_write_nest_lock to annotate the false dead-lock warning. Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch> Signed-off-by: Dennis Li <Dennis.Li@amd.com>

On some EFI systems, the video BIOS is provided by the EFI firmware. The boot stub code stores the physical address of the ROM image in pdev->rom. Currently we attempt to access this pointer using phys_to_virt(), which doesn't work with CONFIG_HIGHMEM. On these systems, attempting to load the radeon module on a x86_32 kernel can result in the following: BUG: unable to handle page fault for address: 3e8ed03c #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page *pde = 00000000 Oops: 0000 [#1] PREEMPT SMP CPU: 0 PID: 317 Comm: systemd-udevd Not tainted 5.6.0-rc3-next-20200228 #2 Hardware name: Apple Computer, Inc. MacPro1,1/Mac-F4208DC8, BIOS MP11.88Z.005C.B08.0707021221 07/02/07 EIP: radeon_get_bios+0x5ed/0xe50 [radeon] Code: 00 00 84 c0 0f 85 12 fd ff ff c7 87 64 01 00 00 00 00 00 00 8b 47 08 8b 55 b0 e8 1e 83 e1 d6 85 c0 74 1a 8b 55 c0 85 d2 74 13 <80> 38 55 75 0e 80 78 01 aa 0f 84 a4 03 00 00 8d 74 26 00 68 dc 06 EAX: 3e8ed03c EBX: 00000000 ECX: 3e8ed03c EDX: 00010000 ESI: 00040000 EDI: eec04000 EBP: eef3fc60 ESP: eef3fbe0 DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00010206 CR0: 80050033 CR2: 3e8ed03c CR3: 2ec77000 CR4: 000006d0 Call Trace: r520_init+0x26/0x240 [radeon] radeon_device_init+0x533/0xa50 [radeon] radeon_driver_load_kms+0x80/0x220 [radeon] drm_dev_register+0xa7/0x180 [drm] radeon_pci_probe+0x10f/0x1a0 [radeon] pci_device_probe+0xd4/0x140 Fix the issue by updating all drivers which can access a platform provided ROM. Instead of calling the helper function pci_platform_rom() which uses phys_to_virt(), call ioremap() directly on the pdev->rom. radeon_read_platform_bios() previously directly accessed an __iomem pointer. Avoid this by calling memcpy_fromio() instead of kmemdup(). pci_platform_rom() now has no remaining callers, so remove it. Link: https://lore.kernel.org/r/20200319021623.5426-1-mikel@mikelr.com Signed-off-by: Mikel Rychliski <mikel@mikelr.com> Signed-off-by: Bjorn Helgaas <bhelgaas@google.com> Acked-by: Alex Deucher <alexander.deucher@amd.com>

In mmu release, we schedule a work to free amn. however there is race between exit_mmap and oom_reap_task_mm. exit_mmap -> mmu_notifier_release oom_reap_task_mm -> __oom_reap_task_mm -> mmu_notifier_invalidate_range_start_nonblock So the amn might have been freed. sync rcu in destroy to wait for ongoing range invalidate. calltrace: [ 4407.908455] BUG: kernel NULL pointer dereference, address: 0000000000000050 [ 4407.915591] #PF: supervisor read access in kernel mode [ 4407.920827] #PF: error_code(0x0000) - not-present page [ 4407.926079] PGD 0 P4D 0 [ 4407.928662] Oops: 0000 [#1] SMP PTI [ 4407.932216] CPU: 3 PID: 55 Comm: oom_reaper Tainted: G W O 5.4.0-rc7+ #1 [ 4407.940206] Hardware name: System manufacturer System Product Name/Z170-A, BIOS 1702 01/28/2016 [ 4407.949080] RIP: 0010:mark_lock+0xc9/0x540 [ 4407.953282] Code: 31 c0 eb 12 48 8d 14 80 48 8d 04 50 48 c1 e0 04 48 05 00 dd 01 9c 41 bc 01 00 00 00 89 d9 41 bf 01 00 00 00 41 d3 e4 4d 63 e4 <4c> 85 60 50 0f 85 50 ff ff ff e8 18 bb ff ff 85 c0 0f 84 40 ff ff [ 4407.972385] RSP: 0018:ffffab4440253ab0 EFLAGS: 00010006 [ 4407.977723] RAX: 0000000000000000 RBX: 0000000000000008 RCX: 0000000000000008 [ 4407.984977] RDX: ffff9df1d9fd8040 RSI: 0000000000000001 RDI: ffffffff9a318d0a [ 4407.992222] RBP: ffffab4440253af0 R08: 0000000000000000 R09: 000000000003b540 [ 4407.999493] R10: 0000000000000000 R11: 0000000000000037 R12: 0000000000000100 [ 4408.006774] R13: ffff9df1d9fd8040 R14: ffff9df1d9fd8c88 R15: 0000000000000001 [ 4408.014062] FS: 0000000000000000(0000) GS:ffff9df1de180000(0000) knlGS:0000000000000000 [ 4408.022295] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4408.028154] CR2: 0000000000000050 CR3: 0000000393410001 CR4: 00000000003606e0 [ 4408.035427] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4408.042689] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4408.049968] Call Trace: [ 4408.052467] __lock_acquire+0x261/0x1600 [ 4408.056486] ? __lock_acquire+0x43a/0x1600 [ 4408.060628] ? __lock_acquire+0x43a/0x1600 [ 4408.064816] lock_acquire+0xb8/0x1c0 [ 4408.068482] ? rwsem_down_read_slowpath+0x1e8/0x5f0 [ 4408.073509] _raw_spin_lock_irq+0x3b/0x50 [ 4408.077624] ? rwsem_down_read_slowpath+0x1e8/0x5f0 [ 4408.082582] rwsem_down_read_slowpath+0x1e8/0x5f0 [ 4408.087401] ? finish_task_switch+0x63/0x230 [ 4408.091759] ? __schedule+0x2b3/0x860 [ 4408.095487] down_read_non_owner+0x86/0x160 [ 4408.099767] ? down_read_non_owner+0x86/0x160 [ 4408.104295] amdgpu_mn_read_lock+0x9f/0xb0 [amdgpu] [ 4408.109364] amdgpu_mn_invalidate_range_start_gfx+0x3f/0x1e0 [amdgpu] [ 4408.115941] __mmu_notifier_invalidate_range_start+0x9e/0x190 [ 4408.121816] ? __oom_reap_task_mm+0x6d/0x220 [ 4408.126166] __oom_reap_task_mm+0x1b5/0x220 [ 4408.130450] oom_reaper+0x4d0/0x650 [ 4408.133991] ? __kthread_parkme+0x2f/0x90 [ 4408.138083] ? finish_wait+0x90/0x90 [ 4408.141715] kthread+0x12c/0x150 [ 4408.145043] ? __oom_reap_task_mm+0x220/0x220 [ 4408.149461] ? kthread_park+0x90/0x90 [ 4408.153208] ret_from_fork+0x3a/0x50 There is another deadlock. calltrace: [ 1635.072660] BUG: sleeping function called from invalid context at ../kernel/locking/rwsem.c:1621 [ 1635.081870] in_atomic(): 0, irqs_disabled(): 0, non_block: 1, pid: 55, name: oom_reaper [ 1635.090106] 4 locks held by oom_reaper/55: [ 1635.091485] init_user_pages: Failed to get user pages: -512 [ 1635.094302] #0: ffff9ca48e94b5d8 (&mm->mmap_sem#2){++++}, at: oom_reaper+0xa4/0x650 [ 1635.108116] #1: ffffffff82750fc0 (mmu_notifier_invalidate_range_start){+.+.}, at: __oom_reap_task_mm+0x6d/0x220 [ 1635.118558] #2: ffffffff827637f0 (srcu){....}, at: __mmu_notifier_invalidate_range_start+0x5/0x190 [ 1635.127879] #3: ffff9ca4f7c605f0 (&amn->read_lock){+.+.}, at: amdgpu_mn_read_lock+0x75/0xb0 [amdgpu] [ 1635.137614] CPU: 3 PID: 55 Comm: oom_reaper Tainted: G W O 5.4.0-rc7+ #1 [ 1635.145787] Hardware name: System manufacturer System Product Name/Z170-A, BIOS 1702 01/28/2016 [ 1635.154744] Call Trace: [ 1635.157264] dump_stack+0x98/0xd5 [ 1635.160688] ___might_sleep+0x175/0x260 [ 1635.164675] __might_sleep+0x4a/0x80 [ 1635.168340] down_read_non_owner+0x20/0x160 [ 1635.172748] amdgpu_mn_read_lock+0x9f/0xb0 [amdgpu] [ 1635.177884] amdgpu_mn_invalidate_range_start_hsa+0x3f/0x180 [amdgpu] [ 1635.184477] __mmu_notifier_invalidate_range_start+0x9e/0x190 [ 1635.190337] ? __oom_reap_task_mm+0x6d/0x220 [ 1635.194725] __oom_reap_task_mm+0x1b5/0x220 [ 1635.199069] oom_reaper+0x4d0/0x650 [ 1635.202611] ? __kthread_parkme+0x2f/0x90 [ 1635.206740] ? finish_wait+0x90/0x90 [ 1635.210425] kthread+0x12c/0x150 [ 1635.213714] ? __oom_reap_task_mm+0x220/0x220 [ 1635.218215] ? kthread_park+0x90/0x90 [ 1635.221976] ret_from_fork+0x3a/0x50 [ 1815.108088] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1815.116065] kworker/1:1 D 0 9357 2 0x80004000 [ 1815.121873] Workqueue: events amdgpu_mn_destroy [amdgpu] [ 1815.127264] Call Trace: [ 1815.129765] __schedule+0x2ab/0x860 [ 1815.133299] ? rwsem_down_write_slowpath+0x329/0x660 [ 1815.138382] schedule+0x3a/0xc0 [ 1815.141613] rwsem_down_write_slowpath+0x32e/0x660 [ 1815.146523] down_write+0x74/0x80 [ 1815.149921] ? down_write+0x40/0x80 [ 1815.153490] ? down_write+0x74/0x80 [ 1815.157285] amdgpu_mn_destroy+0x6e/0x240 [amdgpu] [ 1815.162176] process_one_work+0x231/0x5c0 [ 1815.166313] worker_thread+0x3f/0x3b0 [ 1815.170100] ? __kthread_parkme+0x61/0x90 [ 1815.174204] kthread+0x12c/0x150 [ 1815.177573] ? process_one_work+0x5c0/0x5c0 [ 1815.181857] ? kthread_park+0x90/0x90 [ 1815.185591] ret_from_fork+0x3a/0x50 oom killer want to invalidate range in nonblock context. But the amdgpu_mn_read_lock might sleep, and casue deadlock then. Reviewed-by: Flora Cui <flora.cui@amd.com> Signed-off-by: xinhui pan <xinhui.pan@amd.com> Signed-off-by: Yifan Zhang <Yifan1.Zhang@amd.com>

When amdgpu_ib_ring_tests failed, the reset logic called amdgpu_device_ip_suspend twice, then deadlock occurred. Deadlock log: [ 805.655192] amdgpu 0000:04:00.0: amdgpu: ib ring test failed (-110). [ 806.290952] [drm] free PSP TMR buffer [ 806.319406] ============================================ [ 806.320315] WARNING: possible recursive locking detected [ 806.321225] 5.11.0-custom #1 Tainted: G W OEL [ 806.322135] -------------------------------------------- [ 806.323043] cat/2593 is trying to acquire lock: [ 806.323825] ffff888136b1cdc8 (&adev->dm.dc_lock){+.+.}-{3:3}, at: dm_suspend+0xb8/0x1d0 [amdgpu] [ 806.325668] but task is already holding lock: [ 806.326664] ffff888136b1cdc8 (&adev->dm.dc_lock){+.+.}-{3:3}, at: dm_suspend+0xb8/0x1d0 [amdgpu] [ 806.328430] other info that might help us debug this: [ 806.329539] Possible unsafe locking scenario: [ 806.330549] CPU0 [ 806.330983] ---- [ 806.331416] lock(&adev->dm.dc_lock); [ 806.332086] lock(&adev->dm.dc_lock); [ 806.332738] *** DEADLOCK *** [ 806.333747] May be due to missing lock nesting notation [ 806.334899] 3 locks held by cat/2593: [ 806.335537] #0: ffff888100d3f1b8 (&attr->mutex){+.+.}-{3:3}, at: simple_attr_read+0x4e/0x110 [ 806.337009] #1: ffff888136b1fd78 (&adev->reset_sem){++++}-{3:3}, at: amdgpu_device_lock_adev+0x42/0x94 [amdgpu] [ 806.339018] #2: ffff888136b1cdc8 (&adev->dm.dc_lock){+.+.}-{3:3}, at: dm_suspend+0xb8/0x1d0 [amdgpu] [ 806.340869] stack backtrace: [ 806.341621] CPU: 6 PID: 2593 Comm: cat Tainted: G W OEL 5.11.0-custom #1 [ 806.342921] Hardware name: AMD Celadon-CZN/Celadon-CZN, BIOS WLD0C23N_Weekly_20_12_2 12/23/2020 [ 806.344413] Call Trace: [ 806.344849] dump_stack+0x93/0xbd [ 806.345435] __lock_acquire.cold+0x18a/0x2cf [ 806.346179] lock_acquire+0xca/0x390 [ 806.346807] ? dm_suspend+0xb8/0x1d0 [amdgpu] [ 806.347813] __mutex_lock+0x9b/0x930 [ 806.348454] ? dm_suspend+0xb8/0x1d0 [amdgpu] [ 806.349434] ? amdgpu_device_indirect_rreg+0x58/0x70 [amdgpu] [ 806.350581] ? _raw_spin_unlock_irqrestore+0x47/0x50 [ 806.351437] ? dm_suspend+0xb8/0x1d0 [amdgpu] [ 806.352437] ? rcu_read_lock_sched_held+0x4f/0x80 [ 806.353252] ? rcu_read_lock_sched_held+0x4f/0x80 [ 806.354064] mutex_lock_nested+0x1b/0x20 [ 806.354747] ? mutex_lock_nested+0x1b/0x20 [ 806.355457] dm_suspend+0xb8/0x1d0 [amdgpu] [ 806.356427] ? soc15_common_set_clockgating_state+0x17d/0x19 [amdgpu] [ 806.357736] amdgpu_device_ip_suspend_phase1+0x78/0xd0 [amdgpu] [ 806.360394] amdgpu_device_ip_suspend+0x21/0x70 [amdgpu] [ 806.362926] amdgpu_device_pre_asic_reset+0xb3/0x270 [amdgpu] [ 806.365560] amdgpu_device_gpu_recover.cold+0x679/0x8eb [amdgpu] Signed-off-by: Lang Yu <Lang.Yu@amd.com> Acked-by: Christian KÃnig <christian.koenig@amd.com> Reviewed-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>

In mmu release, we schedule a work to free amn. however there is race between exit_mmap and oom_reap_task_mm. exit_mmap -> mmu_notifier_release oom_reap_task_mm -> __oom_reap_task_mm -> mmu_notifier_invalidate_range_start_nonblock So the amn might have been freed. sync rcu in destroy to wait for ongoing range invalidate. calltrace: [ 4407.908455] BUG: kernel NULL pointer dereference, address: 0000000000000050 [ 4407.915591] #PF: supervisor read access in kernel mode [ 4407.920827] #PF: error_code(0x0000) - not-present page [ 4407.926079] PGD 0 P4D 0 [ 4407.928662] Oops: 0000 [#1] SMP PTI [ 4407.932216] CPU: 3 PID: 55 Comm: oom_reaper Tainted: G W O 5.4.0-rc7+ #1 [ 4407.940206] Hardware name: System manufacturer System Product Name/Z170-A, BIOS 1702 01/28/2016 [ 4407.949080] RIP: 0010:mark_lock+0xc9/0x540 [ 4407.953282] Code: 31 c0 eb 12 48 8d 14 80 48 8d 04 50 48 c1 e0 04 48 05 00 dd 01 9c 41 bc 01 00 00 00 89 d9 41 bf 01 00 00 00 41 d3 e4 4d 63 e4 <4c> 85 60 50 0f 85 50 ff ff ff e8 18 bb ff ff 85 c0 0f 84 40 ff ff [ 4407.972385] RSP: 0018:ffffab4440253ab0 EFLAGS: 00010006 [ 4407.977723] RAX: 0000000000000000 RBX: 0000000000000008 RCX: 0000000000000008 [ 4407.984977] RDX: ffff9df1d9fd8040 RSI: 0000000000000001 RDI: ffffffff9a318d0a [ 4407.992222] RBP: ffffab4440253af0 R08: 0000000000000000 R09: 000000000003b540 [ 4407.999493] R10: 0000000000000000 R11: 0000000000000037 R12: 0000000000000100 [ 4408.006774] R13: ffff9df1d9fd8040 R14: ffff9df1d9fd8c88 R15: 0000000000000001 [ 4408.014062] FS: 0000000000000000(0000) GS:ffff9df1de180000(0000) knlGS:0000000000000000 [ 4408.022295] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4408.028154] CR2: 0000000000000050 CR3: 0000000393410001 CR4: 00000000003606e0 [ 4408.035427] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4408.042689] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4408.049968] Call Trace: [ 4408.052467] __lock_acquire+0x261/0x1600 [ 4408.056486] ? __lock_acquire+0x43a/0x1600 [ 4408.060628] ? __lock_acquire+0x43a/0x1600 [ 4408.064816] lock_acquire+0xb8/0x1c0 [ 4408.068482] ? rwsem_down_read_slowpath+0x1e8/0x5f0 [ 4408.073509] _raw_spin_lock_irq+0x3b/0x50 [ 4408.077624] ? rwsem_down_read_slowpath+0x1e8/0x5f0 [ 4408.082582] rwsem_down_read_slowpath+0x1e8/0x5f0 [ 4408.087401] ? finish_task_switch+0x63/0x230 [ 4408.091759] ? __schedule+0x2b3/0x860 [ 4408.095487] down_read_non_owner+0x86/0x160 [ 4408.099767] ? down_read_non_owner+0x86/0x160 [ 4408.104295] amdgpu_mn_read_lock+0x9f/0xb0 [amdgpu] [ 4408.109364] amdgpu_mn_invalidate_range_start_gfx+0x3f/0x1e0 [amdgpu] [ 4408.115941] __mmu_notifier_invalidate_range_start+0x9e/0x190 [ 4408.121816] ? __oom_reap_task_mm+0x6d/0x220 [ 4408.126166] __oom_reap_task_mm+0x1b5/0x220 [ 4408.130450] oom_reaper+0x4d0/0x650 [ 4408.133991] ? __kthread_parkme+0x2f/0x90 [ 4408.138083] ? finish_wait+0x90/0x90 [ 4408.141715] kthread+0x12c/0x150 [ 4408.145043] ? __oom_reap_task_mm+0x220/0x220 [ 4408.149461] ? kthread_park+0x90/0x90 [ 4408.153208] ret_from_fork+0x3a/0x50 There is another deadlock. calltrace: [ 1635.072660] BUG: sleeping function called from invalid context at ../kernel/locking/rwsem.c:1621 [ 1635.081870] in_atomic(): 0, irqs_disabled(): 0, non_block: 1, pid: 55, name: oom_reaper [ 1635.090106] 4 locks held by oom_reaper/55: [ 1635.091485] init_user_pages: Failed to get user pages: -512 [ 1635.094302] #0: ffff9ca48e94b5d8 (&mm->mmap_sem#2){++++}, at: oom_reaper+0xa4/0x650 [ 1635.108116] #1: ffffffff82750fc0 (mmu_notifier_invalidate_range_start){+.+.}, at: __oom_reap_task_mm+0x6d/0x220 [ 1635.118558] #2: ffffffff827637f0 (srcu){....}, at: __mmu_notifier_invalidate_range_start+0x5/0x190 [ 1635.127879] #3: ffff9ca4f7c605f0 (&amn->read_lock){+.+.}, at: amdgpu_mn_read_lock+0x75/0xb0 [amdgpu] [ 1635.137614] CPU: 3 PID: 55 Comm: oom_reaper Tainted: G W O 5.4.0-rc7+ #1 [ 1635.145787] Hardware name: System manufacturer System Product Name/Z170-A, BIOS 1702 01/28/2016 [ 1635.154744] Call Trace: [ 1635.157264] dump_stack+0x98/0xd5 [ 1635.160688] ___might_sleep+0x175/0x260 [ 1635.164675] __might_sleep+0x4a/0x80 [ 1635.168340] down_read_non_owner+0x20/0x160 [ 1635.172748] amdgpu_mn_read_lock+0x9f/0xb0 [amdgpu] [ 1635.177884] amdgpu_mn_invalidate_range_start_hsa+0x3f/0x180 [amdgpu] [ 1635.184477] __mmu_notifier_invalidate_range_start+0x9e/0x190 [ 1635.190337] ? __oom_reap_task_mm+0x6d/0x220 [ 1635.194725] __oom_reap_task_mm+0x1b5/0x220 [ 1635.199069] oom_reaper+0x4d0/0x650 [ 1635.202611] ? __kthread_parkme+0x2f/0x90 [ 1635.206740] ? finish_wait+0x90/0x90 [ 1635.210425] kthread+0x12c/0x150 [ 1635.213714] ? __oom_reap_task_mm+0x220/0x220 [ 1635.218215] ? kthread_park+0x90/0x90 [ 1635.221976] ret_from_fork+0x3a/0x50 [ 1815.108088] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1815.116065] kworker/1:1 D 0 9357 2 0x80004000 [ 1815.121873] Workqueue: events amdgpu_mn_destroy [amdgpu] [ 1815.127264] Call Trace: [ 1815.129765] __schedule+0x2ab/0x860 [ 1815.133299] ? rwsem_down_write_slowpath+0x329/0x660 [ 1815.138382] schedule+0x3a/0xc0 [ 1815.141613] rwsem_down_write_slowpath+0x32e/0x660 [ 1815.146523] down_write+0x74/0x80 [ 1815.149921] ? down_write+0x40/0x80 [ 1815.153490] ? down_write+0x74/0x80 [ 1815.157285] amdgpu_mn_destroy+0x6e/0x240 [amdgpu] [ 1815.162176] process_one_work+0x231/0x5c0 [ 1815.166313] worker_thread+0x3f/0x3b0 [ 1815.170100] ? __kthread_parkme+0x61/0x90 [ 1815.174204] kthread+0x12c/0x150 [ 1815.177573] ? process_one_work+0x5c0/0x5c0 [ 1815.181857] ? kthread_park+0x90/0x90 [ 1815.185591] ret_from_fork+0x3a/0x50 oom killer want to invalidate range in nonblock context. But the amdgpu_mn_read_lock might sleep, and casue deadlock then. Reviewed-by: Flora Cui <flora.cui@amd.com> Signed-off-by: xinhui pan <xinhui.pan@amd.com> Signed-off-by: Yifan Zhang <Yifan1.Zhang@amd.com>

Noticed the below warning while running a pytorch workload on vega10 GPUs. Change to trylock to avoid conflicts with already held reservation locks. [ +0.000003] WARNING: possible recursive locking detected [ +0.000003] 5.13.0-kfd-rajneesh #1030 Not tainted [ +0.000004] -------------------------------------------- [ +0.000002] python/4822 is trying to acquire lock: [ +0.000004] ffff932cd9a259f8 (reservation_ww_class_mutex){+.+.}-{3:3}, at: amdgpu_bo_release_notify+0xc4/0x160 [amdgpu] [ +0.000203] but task is already holding lock: [ +0.000003] ffff932cbb7181f8 (reservation_ww_class_mutex){+.+.}-{3:3}, at: ttm_eu_reserve_buffers+0x270/0x470 [ttm] [ +0.000017] other info that might help us debug this: [ +0.000002] Possible unsafe locking scenario: [ +0.000003] CPU0 [ +0.000002] ---- [ +0.000002] lock(reservation_ww_class_mutex); [ +0.000004] lock(reservation_ww_class_mutex); [ +0.000003] *** DEADLOCK *** [ +0.000002] May be due to missing lock nesting notation [ +0.000003] 7 locks held by python/4822: [ +0.000003] #0: ffff932c4ac028d0 (&process->mutex){+.+.}-{3:3}, at: kfd_ioctl_map_memory_to_gpu+0x10b/0x320 [amdgpu] [ +0.000232] #1: ffff932c55e830a8 (&info->lock#2){+.+.}-{3:3}, at: amdgpu_amdkfd_gpuvm_map_memory_to_gpu+0x64/0xf60 [amdgpu] [ +0.000241] #2: ffff932cc45b5e68 (&(*mem)->lock){+.+.}-{3:3}, at: amdgpu_amdkfd_gpuvm_map_memory_to_gpu+0xdf/0xf60 [amdgpu] [ +0.000236] #3: ffffb2b35606fd28 (reservation_ww_class_acquire){+.+.}-{0:0}, at: amdgpu_amdkfd_gpuvm_map_memory_to_gpu+0x232/0xf60 [amdgpu] [ +0.000235] #4: ffff932cbb7181f8 (reservation_ww_class_mutex){+.+.}-{3:3}, at: ttm_eu_reserve_buffers+0x270/0x470 [ttm] [ +0.000015] #5: ffffffffc045f700 (*(sspp++)){....}-{0:0}, at: drm_dev_enter+0x5/0xa0 [drm] [ +0.000038] #6: ffff932c52da7078 (&vm->eviction_lock){+.+.}-{3:3}, at: amdgpu_vm_bo_update_mapping+0xd5/0x4f0 [amdgpu] [ +0.000195] stack backtrace: [ +0.000003] CPU: 11 PID: 4822 Comm: python Not tainted 5.13.0-kfd-rajneesh #1030 [ +0.000005] Hardware name: GIGABYTE MZ01-CE0-00/MZ01-CE0-00, BIOS F02 08/29/2018 [ +0.000003] Call Trace: [ +0.000003] dump_stack+0x6d/0x89 [ +0.000010] __lock_acquire+0xb93/0x1a90 [ +0.000009] lock_acquire+0x25d/0x2d0 [ +0.000005] ? amdgpu_bo_release_notify+0xc4/0x160 [amdgpu] [ +0.000184] ? lock_is_held_type+0xa2/0x110 [ +0.000006] ? amdgpu_bo_release_notify+0xc4/0x160 [amdgpu] [ +0.000184] __ww_mutex_lock.constprop.17+0xca/0x1060 [ +0.000007] ? amdgpu_bo_release_notify+0xc4/0x160 [amdgpu] [ +0.000183] ? lock_release+0x13f/0x270 [ +0.000005] ? lock_is_held_type+0xa2/0x110 [ +0.000006] ? amdgpu_bo_release_notify+0xc4/0x160 [amdgpu] [ +0.000183] amdgpu_bo_release_notify+0xc4/0x160 [amdgpu] [ +0.000185] ttm_bo_release+0x4c6/0x580 [ttm] [ +0.000010] amdgpu_bo_unref+0x1a/0x30 [amdgpu] [ +0.000183] amdgpu_vm_free_table+0x76/0xa0 [amdgpu] [ +0.000189] amdgpu_vm_free_pts+0xb8/0xf0 [amdgpu] [ +0.000189] amdgpu_vm_update_ptes+0x411/0x770 [amdgpu] [ +0.000191] amdgpu_vm_bo_update_mapping+0x324/0x4f0 [amdgpu] [ +0.000191] amdgpu_vm_bo_update+0x251/0x610 [amdgpu] [ +0.000191] update_gpuvm_pte+0xcc/0x290 [amdgpu] [ +0.000229] ? amdgpu_vm_bo_map+0xd7/0x130 [amdgpu] [ +0.000190] amdgpu_amdkfd_gpuvm_map_memory_to_gpu+0x912/0xf60 [amdgpu] [ +0.000234] kfd_ioctl_map_memory_to_gpu+0x182/0x320 [amdgpu] [ +0.000218] kfd_ioctl+0x2b9/0x600 [amdgpu] [ +0.000216] ? kfd_ioctl_unmap_memory_from_gpu+0x270/0x270 [amdgpu] [ +0.000216] ? lock_release+0x13f/0x270 [ +0.000006] ? __fget_files+0x107/0x1e0 [ +0.000007] __x64_sys_ioctl+0x8b/0xd0 [ +0.000007] do_syscall_64+0x36/0x70 [ +0.000004] entry_SYSCALL_64_after_hwframe+0x44/0xae [ +0.000007] RIP: 0033:0x7fbff90a7317 [ +0.000004] Code: b3 66 90 48 8b 05 71 4b 2d 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 41 4b 2d 00 f7 d8 64 89 01 48 [ +0.000005] RSP: 002b:00007fbe301fe648 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ +0.000006] RAX: ffffffffffffffda RBX: 00007fbcc402d820 RCX: 00007fbff90a7317 [ +0.000003] RDX: 00007fbe301fe690 RSI: 00000000c0184b18 RDI: 0000000000000004 [ +0.000003] RBP: 00007fbe301fe690 R08: 0000000000000000 R09: 00007fbcc402d880 [ +0.000003] R10: 0000000002001000 R11: 0000000000000246 R12: 00000000c0184b18 [ +0.000003] R13: 0000000000000004 R14: 00007fbf689593a0 R15: 00007fbcc402d820 Cc: Christian König <christian.koenig@amd.com> Cc: Felix Kuehling <Felix.Kuehling@amd.com> Cc: Alex Deucher <Alexander.Deucher@amd.com> Reviewed-by: Christian König <christian.koenig@amd.com> Reviewed-by: Felix Kuehling <Felix.Kuehling@amd.com> Signed-off-by: Rajneesh Bhardwaj <rajneesh.bhardwaj@amd.com> Change-Id: I963075708afaaadccd6b70ad73358bd89805f257

Removal of DC_FP_* wrappers from dml (9696679) provokes a mutex lock [2] on the amdgpu driver. Re-arrange the dcn10 code to avoid locking the mutex by placing the DC_FP_* wrappers around the proper functions. This fixes the following WARN/stacktrace: BUG: sleeping function called from invalid context at kernel/locking/mutex.c:283 in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 227, name: systemd-udevd preempt_count: 1, expected: 0 CPU: 4 PID: 227 Comm: systemd-udevd Not tainted 6.0.0-rc6-qtec-standard #2 Hardware name: Qtechnology A/S QT5222/QT5221, BIOS v1.0.1 06/07/2021 Call Trace: <TASK> dump_stack_lvl+0x33/0x42 __might_resched.cold.172+0xa5/0xb3 mutex_lock+0x1a/0x40 amdgpu_dpm_get_clock_by_type_with_voltage+0x38/0x70 [amdgpu] dm_pp_get_clock_levels_by_type_with_voltage+0x64/0xa0 [amdgpu] dcn_bw_update_from_pplib+0x70/0x340 [amdgpu] dcn10_create_resource_pool+0x8c8/0xd20 [amdgpu] ? __kmalloc+0x1c7/0x4a0 dc_create_resource_pool+0xe7/0x190 [amdgpu] dc_create+0x212/0x5d0 [amdgpu] amdgpu_dm_init+0x246/0x370 [amdgpu] ? schedule_hrtimeout_range_clock+0x93/0x120 ? phm_wait_for_register_unequal.part.1+0x4a/0x80 [amdgpu] dm_hw_init+0xe/0x20 [amdgpu] amdgpu_device_init.cold.56+0x1324/0x1653 [amdgpu] ? pci_bus_read_config_word+0x43/0x80 amdgpu_driver_load_kms+0x15/0x120 [amdgpu] amdgpu_pci_probe+0x116/0x320 [amdgpu] pci_device_probe+0x97/0x110 really_probe+0xdd/0x340 __driver_probe_device+0x80/0x170 driver_probe_device+0x1f/0x90 __driver_attach+0xdc/0x180 ? __device_attach_driver+0x100/0x100 ? __device_attach_driver+0x100/0x100 bus_for_each_dev+0x74/0xc0 bus_add_driver+0x19e/0x210 ? kset_find_obj+0x30/0xa0 ? 0xffffffffa0a5b000 driver_register+0x6b/0xc0 ? 0xffffffffa0a5b000 do_one_initcall+0x4a/0x1f0 ? __vunmap+0x28e/0x2f0 ? __cond_resched+0x15/0x30 ? kmem_cache_alloc_trace+0x3d/0x440 do_init_module+0x4a/0x1e0 load_module+0x1cba/0x1e10 ? __do_sys_finit_module+0xb7/0x120 __do_sys_finit_module+0xb7/0x120 do_syscall_64+0x3c/0x80 entry_SYSCALL_64_after_hwframe+0x63/0xcd RIP: 0033:0x7ff2b5f5422d Code: 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d c3 ab 0e 00 f7 d8 64 89 01 48 RSP: 002b:00007ffc44ab28e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 RAX: ffffffffffffffda RBX: 0000555c566a9240 RCX: 00007ff2b5f5422d RDX: 0000000000000000 RSI: 00007ff2b60bb353 RDI: 0000000000000019 RBP: 00007ff2b60bb353 R08: 0000000000000000 R09: 0000555c566a9240 R10: 0000000000000019 R11: 0000000000000246 R12: 0000000000000000 R13: 0000000000020000 R14: 0000000000000000 R15: 0000000000000000 </TASK> Fixes: 9696679 ("drm/amd/display: remove DC_FP_* wrapper from dml folder") Reviewed-by: Hamza Mahfooz <hamza.mahfooz@amd.com> Tested-by: Daniel Wheeler <daniel.wheeler@amd.com> Signed-off-by: Daniel Gomez <daniel@qtec.com>

Clear adev->in_suspend flag when fails to suspend, otherwise it will cause too much warnings like: [ 1802.212027] ------------[ cut here ]------------ [ 1802.212028] WARNING: CPU: 97 PID: 11282 at drivers/gpu/drm/amd/amdgpu/amdgpu_object.c:452 amdgpu_bo_free_kernel+0xf9/0x120 [amdgpu] [ 1802.212198] Modules linked in: amdgpu(E-) tcp_diag(E) inet_diag(E) rfkill(E) intel_rapl_msr(E) intel_rapl_common(E) intel_uncore_frequency(E) intel_uncore_frequency_common(E) i10nm_edac(E) nfit(E) x86_pkg_temp_thermal(E) intel_powerclamp(E) coretemp(E) kvm_intel(E) kvm(E) crct10dif_pclmul(E) crc32_pclmul(E) ghash_clmulni_intel(E) rapl(E) iTCO_wdt(E) pmt_telemetry(E) iTCO_vendor_support(E) pmt_class(E) intel_cstate(E) snd_hda_intel(E) ipmi_ssif(E) snd_intel_dspcfg(E) snd_hda_codec(E) snd_hda_core(E) snd_hwdep(E) snd_pcm(E) snd_timer(E) intel_uncore(E) snd(E) cdc_ether(E) pcspkr(E) i2c_i801(E) idxd(E) mei_me(E) usbnet(E) ses(E) isst_if_mmio(E) isst_if_mbox_pci(E) mii(E) joydev(E) soundcore(E) isst_if_common(E) idxd_bus(E) mei(E) enclosure(E) intel_vsec(E) i2c_smbus(E) i2c_ismt(E) sunrpc(E) acpi_power_meter(E) ipmi_si(E) acpi_ipmi(E) ipmi_devintf(E) acpi_pad(E) ipmi_msghandler(E) vfat(E) fat(E) sg(E) video(E) amdxcp(E) drm_ttm_helper(E) ttm(E) drm_exec(E) gpu_sched(E) drm_suballoc_helper(E) crc32c_intel(E) drm_buddy(E) [ 1802.212223] ast(E) drm_shmem_helper(E) drm_display_helper(E) i2c_algo_bit(E) drm_kms_helper(E) virtio_net(E) mpt3sas(E) drm(E) net_failover(E) ahci(E) raid_class(E) failover(E) libahci(E) scsi_transport_sas(E) dimlib(E) libata(E) i2c_core(E) wmi(E) pinctrl_emmitsburg(E) [last unloaded: amdgpu(E)] [ 1802.212231] CPU: 97 PID: 11282 Comm: rmmod Kdump: loaded Tainted: G W E 6.10.0+ ROCm#2 [ 1802.212232] Hardware name: Alibaba Alibaba Cloud ECS/Alibaba Cloud ECS, BIOS 3.0.ES.AL.P.087.05 04/07/2024 [ 1802.212232] RIP: 0010:amdgpu_bo_free_kernel+0xf9/0x120 [amdgpu] [ 1802.212401] Code: 00 00 00 4d 85 e4 74 08 49 c7 04 24 00 00 00 00 48 85 ed 74 08 48 c7 45 00 00 00 00 00 5b 5d 41 5c 41 5d 41 5e c3 cc cc cc cc <0f> 0b e9 3b ff ff ff 3d 00 fe ff ff 74 b3 49 8b be b0 10 ff ff 4c [ 1802.212402] RSP: 0018:ffffbe0e92087cb0 EFLAGS: 00010202 [ 1802.212403] RAX: 0000000000000206 RBX: ffff9cbc06c37f10 RCX: 000000000020000d [ 1802.212404] RDX: ffff9cbc06c37f18 RSI: ffff9cbc06c37f58 RDI: ffff9cbc06c37f10 [ 1802.212407] RBP: ffff9cbc06c37f18 R08: ffff9cba42388800 R09: 000000000020000d [ 1802.212409] R10: 0000000000040000 R11: 0000000000000006 R12: ffff9cbc06c37f58 [ 1802.212410] R13: ffff9cba4238dc00 R14: ffff9cbc06c0ef50 R15: ffff9cbc06c00000 [ 1802.212411] FS: 00007f927dc4e740(0000) GS:ffff9db47e880000(0000) knlGS:0000000000000000 [ 1802.212412] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1802.212413] CR2: 000056342b2be1a8 CR3: 00000003f1546005 CR4: 0000000000f70ef0 [ 1802.212413] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1802.212414] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400 [ 1802.212415] PKRU: 55555554 [ 1802.212415] Call Trace: [ 1802.212416] <TASK> [ 1802.212417] ? __warn+0x83/0x130 [ 1802.212419] ? amdgpu_bo_free_kernel+0xf9/0x120 [amdgpu] [ 1802.212586] ? __report_bug+0xea/0x100 [ 1802.212588] ? report_bug+0x24/0x70 [ 1802.212589] ? handle_bug+0x3c/0x70 [ 1802.212590] ? exc_invalid_op+0x18/0x70 [ 1802.212591] ? asm_exc_invalid_op+0x1a/0x20 [ 1802.212594] ? amdgpu_bo_free_kernel+0xf9/0x120 [amdgpu] [ 1802.212746] amdgpu_ring_fini+0x91/0x120 [amdgpu] [ 1802.212901] amdgpu_jpeg_sw_fini+0xb2/0xe0 [amdgpu] [ 1802.213106] amdgpu_device_ip_fini.isra.0+0xb1/0x1c0 [amdgpu] [ 1802.213247] amdgpu_device_fini_sw+0x49/0x290 [amdgpu] [ 1802.213413] amdgpu_driver_release_kms+0x16/0x30 [amdgpu] [ 1802.213576] devm_drm_dev_init_release+0x4e/0x70 [drm] [ 1802.213602] release_nodes+0x35/0xb0 [ 1802.213605] devres_release_all+0x8f/0xd0 [ 1802.213606] device_unbind_cleanup+0xe/0x70 [ 1802.213609] device_release_driver_internal+0x1bc/0x200 [ 1802.213611] driver_detach+0x48/0x90 [ 1802.213613] bus_remove_driver+0x6d/0xf0 [ 1802.213615] pci_unregister_driver+0x2e/0xb0 [ 1802.213618] amdgpu_exit+0x15/0x1c4 [amdgpu] [ 1802.213861] __do_sys_delete_module.constprop.0+0x176/0x310 [ 1802.213863] do_syscall_64+0x5d/0x170 [ 1802.213866] entry_SYSCALL_64_after_hwframe+0x76/0x7e [ 1802.213869] RIP: 0033:0x7f927d6620cb [ 1802.213870] Code: 73 01 c3 48 8b 0d a5 6d 19 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 b0 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 75 6d 19 00 f7 d8 64 89 01 48 [ 1802.213871] RSP: 002b:00007fff031d9d78 EFLAGS: 00000206 ORIG_RAX: 00000000000000b0 [ 1802.213873] RAX: ffffffffffffffda RBX: 000055d8d6df69e0 RCX: 00007f927d6620cb [ 1802.213873] RDX: 000000000000000a RSI: 0000000000000800 RDI: 000055d8d6df6a48 [ 1802.213874] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 1802.213875] R10: 00007f927d7aaac0 R11: 0000000000000206 R12: 00007fff031d9fa0 [ 1802.213876] R13: 00007fff031da5f1 R14: 000055d8d6df62a0 R15: 000055d8d6df69e0 [ 1802.213878] </TASK> [ 1802.213879] ---[ end trace 0000000000000000 ]--- Signed-off-by: Jiang Liu <gerry@linux.alibaba.com>

Fix following false warnings: [12372.640824] ------------[ cut here ]------------ [12372.640825] WARNING: CPU: 48 PID: 1529 at drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c:633 amdgpu_irq_put+0x49/0x70 [amdgpu] [12372.641032] Modules linked in: amdgpu(E+) binfmt_misc(E) tcp_diag(E) inet_diag(E) rfkill(E) sunrpc(E) intel_rapl_msr(E) intel_rapl_common(E) intel_uncore_frequency(E) intel_uncore_frequency_common(E) i10nm_edac(E) nfit(E) x86_pkg_temp_thermal(E) intel_powerclamp(E) coretemp(E) kvm_intel(E) kvm(E) ipmi_ssif(E) iTCO_wdt(E) pmt_telemetry(E) iTCO_vendor_support(E) pmt_class(E) crct10dif_pclmul(E) crc32_pclmul(E) ghash_clmulni_intel(E) rapl(E) intel_cstate(E) snd_hda_intel(E) snd_intel_dspcfg(E) snd_hda_codec(E) snd_hda_core(E) snd_hwdep(E) intel_uncore(E) snd_pcm(E) cdc_ether(E) pcspkr(E) snd_timer(E) usbnet(E) snd(E) idxd(E) joydev(E) mii(E) isst_if_mbox_pci(E) ses(E) isst_if_mmio(E) enclosure(E) mei_me(E) isst_if_common(E) soundcore(E) idxd_bus(E) i2c_i801(E) intel_vsec(E) mei(E) i2c_ismt(E) i2c_smbus(E) acpi_power_meter(E) ipmi_si(E) acpi_ipmi(E) ipmi_devintf(E) ipmi_msghandler(E) acpi_pad(E) vfat(E) fat(E) sg(E) video(E) amdxcp(E) drm_ttm_helper(E) ttm(E) drm_exec(E) gpu_sched(E) drm_suballoc_helper(E) ast(E) [12372.641060] drm_buddy(E) drm_shmem_helper(E) drm_display_helper(E) i2c_algo_bit(E) drm_kms_helper(E) virtio_net(E) ahci(E) mpt3sas(E) libahci(E) net_failover(E) crc32c_intel(E) drm(E) failover(E) raid_class(E) libata(E) i2c_core(E) scsi_transport_sas(E) dimlib(E) wmi(E) pinctrl_emmitsburg(E) [last unloaded: amdgpu(E)] [12372.641069] CPU: 48 PID: 1529 Comm: kworker/48:1 Kdump: loaded Tainted: G W E 6.10.0+ ROCm#2 [12372.641070] Hardware name: Alibaba Alibaba Cloud ECS/Alibaba Cloud ECS, BIOS 3.0.ES.AL.P.087.05 04/07/2024 [12372.641071] Workqueue: events work_for_cpu_fn [12372.641073] RIP: 0010:amdgpu_irq_put+0x49/0x70 [amdgpu] [12372.641271] Code: 48 8b 4e 10 48 83 39 00 74 2c 89 d1 48 8d 04 88 8b 08 85 c9 74 14 f0 ff 08 b8 00 00 00 00 74 05 c3 cc cc cc cc e9 57 fd ff ff <0f> 0b b8 ea ff ff ff c3 cc cc cc cc b8 ea ff ff ff c3 cc cc cc cc [12372.641273] RSP: 0018:ffffb570dd85bd70 EFLAGS: 00010246 [12372.641274] RAX: ffff97b0874a0328 RBX: ffff97b7c9145650 RCX: 0000000000000000 [12372.641275] RDX: 0000000000000000 RSI: ffff97b7c9116620 RDI: ffff97b7c9100000 [12372.641276] RBP: ffff97b7c9100000 R08: 0000000000000000 R09: 0000000000000000 [12372.641277] R10: 0000000000072963 R11: ffff98abbce33be4 R12: ffff97b7c9145638 [12372.641278] R13: ffffffffc13ae9c0 R14: ffff97b7c9100000 R15: ffff97b7c9100010 [12372.641279] FS: 0000000000000000(0000) GS:ffff98abbce00000(0000) knlGS:0000000000000000 [12372.641280] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12372.641281] CR2: 00005599a599d120 CR3: 000001043d292001 CR4: 0000000000f70ef0 [12372.641281] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12372.641282] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400 [12372.641283] PKRU: 55555554 [12372.641284] Call Trace: [12372.641284] <TASK> [12372.641285] ? __warn+0x83/0x130 [12372.641287] ? amdgpu_irq_put+0x49/0x70 [amdgpu] [12372.641487] ? __report_bug+0xea/0x100 [12372.641489] ? report_bug+0x24/0x70 [12372.641490] ? handle_bug+0x3c/0x70 [12372.641491] ? exc_invalid_op+0x18/0x70 [12372.641493] ? asm_exc_invalid_op+0x1a/0x20 [12372.641495] ? amdgpu_irq_put+0x49/0x70 [amdgpu] [12372.641685] soc15_common_hw_fini+0xb2/0xe0 [amdgpu] [12372.641876] amdgpu_device_ip_fini_early.isra.0+0x178/0x200 [amdgpu] [12372.642052] amdgpu_device_fini_hw+0x10b/0x240 [amdgpu] [12372.642236] amdgpu_driver_load_kms+0x80/0xc0 [amdgpu] [12372.642419] amdgpu_pci_probe+0x198/0x420 [amdgpu] [12372.642599] local_pci_probe+0x47/0xa0 [12372.642602] work_for_cpu_fn+0x17/0x30 [12372.642603] process_one_work+0x179/0x390 [12372.642605] worker_thread+0x2e7/0x370 [12372.642607] ? __pfx_worker_thread+0x10/0x10 [12372.642608] kthread+0xd0/0x100 [12372.642610] ? __pfx_kthread+0x10/0x10 [12372.642612] ret_from_fork+0x31/0x50 [12372.642614] ? __pfx_kthread+0x10/0x10 [12372.642616] ret_from_fork_asm+0x1a/0x30 [12372.642618] </TASK> [12372.642618] ---[ end trace 0000000000000000 ]--- Signed-off-by: Jiang Liu <gerry@linux.alibaba.com>

If some GPU device failed to probe, `rmmod amdgpu` will trigger a use after free bug related to amdgpu_driver_release_kms() as: 2024-12-26 16:17:45 [16002.085540] BUG: kernel NULL pointer dereference, address: 0000000000000000 2024-12-26 16:17:45 [16002.093792] #PF: supervisor read access in kernel mode 2024-12-26 16:17:45 [16002.099993] #PF: error_code(0x0000) - not-present page 2024-12-26 16:17:45 [16002.106188] PGD 0 P4D 0 2024-12-26 16:17:45 [16002.109464] Oops: Oops: 0000 [ROCm#1] PREEMPT SMP NOPTI 2024-12-26 16:17:45 [16002.115372] CPU: 2 PID: 14375 Comm: rmmod Kdump: loaded Tainted: G W E 6.10.0+ ROCm#2 2024-12-26 16:17:45 [16002.125577] Hardware name: Alibaba Alibaba Cloud ECS/Alibaba Cloud ECS, BIOS 3.0.ES.AL.P.087.05 04/07/2024 2024-12-26 16:17:45 [16002.136858] RIP: 0010:drm_sched_fini+0x3f/0xe0 [gpu_sched] 2024-12-26 16:17:45 [16002.143463] Code: 60 c6 87 be 00 00 00 01 e8 ce e0 90 d8 48 8d bb 80 00 00 00 e8 c2 e0 90 d8 8b 43 20 85 c0 74 51 45 31 e4 48 8b 43 28 4d 63 ec <4a> 8b 2c e8 48 89 ef e8 f5 0e 59 d9 48 8b 45 10 48 8d 55 10 48 39 2024-12-26 16:17:45 [16002.164992] RSP: 0018:ffffb570dbbb7da8 EFLAGS: 00010246 2024-12-26 16:17:45 [16002.171316] RAX: 0000000000000000 RBX: ffff96b0fdadc878 RCX: 0000000000000000 2024-12-26 16:17:46 [16002.179784] RDX: 000fffffffe00000 RSI: 0000000000000000 RDI: ffff96b0fdadc8f8 2024-12-26 16:17:46 [16002.188252] RBP: ffff96b0fdadc800 R08: ffff97abbd035040 R09: ffffffff9ac52540 2024-12-26 16:17:46 [16002.196722] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 2024-12-26 16:17:46 [16002.205179] R13: 0000000000000000 R14: ffff96b0fdadfc00 R15: 0000000000000000 2024-12-26 16:17:46 [16002.213648] FS: 00007f2737000740(0000) GS:ffff97abbd100000(0000) knlGS:0000000000000000 2024-12-26 16:17:46 [16002.223189] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2024-12-26 16:17:46 [16002.230103] CR2: 0000000000000000 CR3: 000000011be3a005 CR4: 0000000000f70ef0 2024-12-26 16:17:46 [16002.238581] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 2024-12-26 16:17:46 [16002.247053] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400 e024se+0x3c/0x90 [amdxcp] 2024-12-26 16:17:46 [16002.337645] __do_sys_delete_module.constprop.0+0x176/0x310 2024-12-26 16:17:46 [16002.344324] do_syscall_64+0x5d/0x170 2024-12-26 16:17:46 [16002.348858] entry_SYSCALL_64_after_hwframe+0x76/0x7e 2024-12-26 16:17:46 [16002.354956] RIP: 0033:0x7f2736a620cb-12-26 Fix it by unplugging xcp drm devices when failed to probe GPU devices. Signed-off-by: Jiang Liu <gerry@linux.alibaba.com> Tested-by: Shuo Liu <shuox.liu@linux.alibaba.com>

mattmacy closed this as completed Apr 25, 2016

ernstp mentioned this issue Jun 14, 2017

Null pointer dereference in update_stream_scaling_settings #23

Closed

hanbin973 mentioned this issue Jul 6, 2017

Poor Display with kernel #24

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kernel oops when running hip kernel with dev branch ROCR/ROCK #2

kernel oops when running hip kernel with dev branch ROCR/ROCK #2

mattmacy commented Mar 24, 2016

jedwards-AMD commented Apr 4, 2016

mattmacy commented Apr 25, 2016

kernel oops when running hip kernel with dev branch ROCR/ROCK #2

kernel oops when running hip kernel with dev branch ROCR/ROCK #2

Comments

mattmacy commented Mar 24, 2016

jedwards-AMD commented Apr 4, 2016

mattmacy commented Apr 25, 2016