Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Problem with BTRFS causes kernel panic and everything to go read only #1377

Closed
jholz88 opened this issue Jun 8, 2024 · 3 comments
Closed

Problem with BTRFS causes kernel panic and everything to go read only #1377

jholz88 opened this issue Jun 8, 2024 · 3 comments

Comments

@jholz88
Copy link

jholz88 commented Jun 8, 2024

Describe the bug

I updated to the latest version of bluefin dx 40, with asus and nvidia, and starting noticing my system would end up in a read only state. I ran 'dmesg' and found a BTRFS related kernel panic that's causing the filesystem to switch to read only. I've verified with smartctl that the ssd is fine, I used btrfs to scrub and modified fstab to clear cache to no avail. Seems like the extensions and and some buttons are missing since this started. I've included the kernel panic extracted from dmesg. It's usually that specific file, but I've seen other files from that same directory trigger the panic.

What did you expect to happen?

I was hoping I could just rollback then update past the problem, but it's still happening, even after rolling back, and I can't update or rebase without triggering the panic which makes the system read only.

Output of rpm-ostree status

State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: inactive
Deployments:
  ostree-image-signed:docker://ghcr.io/ublue-os/bluefin-dx-asus-nvidia:latest
                   Digest: sha256:98d9e577b369e02ef8be5b4e2596131f4c8967002475080c63e88a585c969737
                  Version: 40.20240607.0 (2024-06-07T17:26:27Z)
                     Diff: 180 upgraded, 8 removed, 18 added
          LayeredPackages: ncurses-compat-libs perl-File-Compare perl-File-Copy perl-FindBin perl-IPC-Cmd perl-JSON-PP perl-Thread-Queue
                           perl-Time-Piece yggdrasil yum-plugin-copr

● ostree-image-signed:docker://ghcr.io/ublue-os/bluefin-dx-asus-nvidia:latest
                   Digest: sha256:775637ac492dfa2f852c4e19248d85d787c249da99563590011f2d363309d032
                  Version: 40.20240531.0 (2024-05-31T16:54:22Z)
          LayeredPackages: ncurses-compat-libs ncurses-devel patch perl-File-Compare perl-File-Copy perl-FindBin perl-IPC-Cmd perl-JSON-PP
                           perl-Thread-Queue perl-Time-Piece yggdrasil yum-plugin-copr

  ostree-image-signed:docker://ghcr.io/ublue-os/bluefin-dx-asus-nvidia:latest
                   Digest: sha256:30494bfee131fdc0e0cf25860e78ab8510890bb85d277aa2134183f2902be3ba
                  Version: 40.20240602.0 (2024-06-03T17:11:59Z)
          LayeredPackages: ncurses-compat-libs ncurses-devel patch perl-File-Compare perl-File-Copy perl-FindBin perl-IPC-Cmd perl-JSON-PP
                           perl-Thread-Queue perl-Time-Piece pmbootstrap yggdrasil yum-plugin-copr

Output of groups

jholz88 wheel

Extra information or context

[ 1394.006485] BTRFS info (device dm-0): failed to delete reference to accessories-dictionary.png, inode 137304 parent 8182597
[ 1394.006493] ------------[ cut here ]------------
[ 1394.006495] BTRFS: Transaction aborted (error -2)
[ 1394.006502] WARNING: CPU: 13 PID: 18548 at fs/btrfs/inode.c:4144 __btrfs_unlink_inode.cold+0x72/0x14e
[ 1394.006507] Modules linked in: uinput snd_seq_dummy snd_hrtimer veth xt_nat xt_conntrack xt_MASQUERADE nf_conntrack_netlink xt_addrtype nft_compat br_netfilter bridge stp llc tun overlay nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables qrtr bnep sunrpc intel_rapl_msr ucsi_ccg snd_sof_amd_acp63 snd_sof_amd_vangogh snd_sof_amd_rembrandt snd_sof_amd_renoir snd_sof_amd_acp snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_hda_codec_realtek amd_atl snd_sof_utils snd_pci_ps snd_hda_codec_hdmi snd_hda_codec_generic snd_amd_sdw_acpi snd_hda_scodec_component intel_rapl_common btusb soundwire_amd btrtl soundwire_generic_allocation soundwire_bus edac_mce_amd snd_soc_core kvm_amd btintel snd_compress ac97_bus asus_nb_wmi snd_hda_intel snd_intel_dspcfg kvm btbcm snd_intel_sdw_acpi snd_hda_codec btmtk wmi_bmof rapl iwlmvm snd_pcm_dmaengine bluetooth
[ 1394.006584] snd_rpl_pci_acp6x pcspkr k10temp wl(POE) binfmt_misc acpi_cpufreq snd_hda_core snd_hwdep mac80211 snd_acp_pci snd_seq snd_acp_legacy_common snd_pci_acp6x libarc4 snd_seq_device snd_pci_acp5x i2c_piix4 snd_rn_pci_acp3x snd_acp_config snd_pcm iwlwifi snd_soc_acpi snd_timer vfat snd i2c_nvidia_gpu fat soundcore snd_pci_acp3x asus_wireless joydev brcmfmac brcmutil cfg80211 mmc_core scsi_dh_rdac scsi_dh_emc scsi_dh_alua kvmfr(OE) loop dm_multipath nfnetlink zram dm_crypt amdgpu amdxcp drm_exec gpu_sched drm_buddy i2c_algo_bit drm_suballoc_helper drm_display_helper cec drm_ttm_helper ccp ttm crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni polyval_generic nvme nvme_core ghash_clmulni_intel sha512_ssse3 hid_multitouch ucsi_acpi sha256_ssse3 typec_ucsi sha1_ssse3 sp5100_tco typec nvme_auth i2c_hid_acpi i2c_hid hid_asus asus_wmi sparse_keymap platform_profile rfkill nvidia_drm(POE) nvidia_modeset(POE) video wmi nvidia_uvm(POE) nvidia(POE) v4l2loopback(OE) videodev mc ip6_tables ip_tables fuse i2c_dev
[ 1394.006672] CPU: 13 PID: 18548 Comm: pool-rpm-ostree Tainted: P W OE 6.9.1-669.rog.fc40.x86_64 #1
[ 1394.006675] Hardware name: ASUSTeK COMPUTER INC. ROG Zephyrus G14 GA401IV_GA401IV/GA401IV, BIOS GA401IV.222 09/28/2023
[ 1394.006677] RIP: 0010:__btrfs_unlink_inode.cold+0x72/0x14e
[ 1394.006679] Code: 89 54 24 20 e8 64 99 ff ff 44 8b 54 24 20 84 c0 0f 84 b9 00 00 00 44 89 d6 48 c7 c7 48 af b8 b3 44 89 54 24 20 e8 93 5f 03 ff <0f> 0b 44 8b 54 24 20 41 b8 01 00 00 00 44 89 d1 41 83 e0 01 ba 30
[ 1394.006681] RSP: 0018:ffff984513abbb88 EFLAGS: 00010282
[ 1394.006684] RAX: 0000000000000000 RBX: ffff88b8eb784d10 RCX: 0000000000000027
[ 1394.006686] RDX: ffff88b99f8a18c8 RSI: 0000000000000001 RDI: ffff88b99f8a18c0
[ 1394.006687] RBP: ffff984513abbbe8 R08: 0000000000000000 R09: 69746361736e6172
[ 1394.006689] R10: 74726f6261206e6f R11: 6f72726528206465 R12: ffff88b958cfc8c8
[ 1394.006690] R13: ffff88b6aca37930 R14: ffff88b692c509d8 R15: ffff88b697fd9800
[ 1394.006692] FS: 00007fa3920006c0(0000) GS:ffff88b99f880000(0000) knlGS:0000000000000000
[ 1394.006694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1394.006696] CR2: 00007fcc7c18c1dc CR3: 0000000133fa6000 CR4: 0000000000350ef0
[ 1394.006698] Call Trace:
[ 1394.006700]
[ 1394.006702] ? __btrfs_unlink_inode.cold+0x72/0x14e
[ 1394.006704] ? __warn.cold+0x8e/0xe8
[ 1394.006707] ? __btrfs_unlink_inode.cold+0x72/0x14e
[ 1394.006713] ? report_bug+0xff/0x140
[ 1394.006716] ? console_unlock+0x84/0x130
[ 1394.006720] ? handle_bug+0x3c/0x80
[ 1394.006723] ? exc_invalid_op+0x17/0x70
[ 1394.006726] ? asm_exc_invalid_op+0x1a/0x20
[ 1394.006731] ? __btrfs_unlink_inode.cold+0x72/0x14e
[ 1394.006734] ? __btrfs_unlink_inode.cold+0x72/0x14e
[ 1394.006738] btrfs_unlink+0xdd/0x150
[ 1394.006743] vfs_unlink+0x110/0x280
[ 1394.006748] do_unlinkat+0x2ae/0x330
[ 1394.006753] __x64_sys_unlinkat+0x56/0xc0
[ 1394.006755] do_syscall_64+0x82/0x160
[ 1394.006759] ? srso_return_thunk+0x5/0x5f
[ 1394.006761] ? do_unlinkat+0xbe/0x330
[ 1394.006765] ? srso_return_thunk+0x5/0x5f
[ 1394.006768] ? syscall_exit_to_user_mode+0x75/0x230
[ 1394.006770] ? srso_return_thunk+0x5/0x5f
[ 1394.006773] ? do_syscall_64+0x8e/0x160
[ 1394.006775] ? srso_return_thunk+0x5/0x5f
[ 1394.006777] ? syscall_exit_to_user_mode+0x75/0x230
[ 1394.006780] ? srso_return_thunk+0x5/0x5f
[ 1394.006782] ? do_syscall_64+0x8e/0x160
[ 1394.006785] ? srso_return_thunk+0x5/0x5f
[ 1394.006787] ? __x64_sys_fcntl+0x81/0xc0
[ 1394.006790] ? srso_return_thunk+0x5/0x5f
[ 1394.006792] ? syscall_exit_to_user_mode+0x75/0x230
[ 1394.006795] ? srso_return_thunk+0x5/0x5f
[ 1394.006797] ? do_syscall_64+0x8e/0x160
[ 1394.006799] ? srso_return_thunk+0x5/0x5f
[ 1394.006801] ? do_syscall_64+0x8e/0x160
[ 1394.006803] ? srso_return_thunk+0x5/0x5f
[ 1394.006806] ? do_syscall_64+0x8e/0x160
[ 1394.006807] ? srso_return_thunk+0x5/0x5f
[ 1394.006810] ? do_syscall_64+0x8e/0x160
[ 1394.006811] ? do_syscall_64+0x8e/0x160
[ 1394.006813] ? srso_return_thunk+0x5/0x5f
[ 1394.006816] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 1394.006819] RIP: 0033:0x7fa3a70126fb
[ 1394.006835] Code: 77 05 c3 0f 1f 40 00 48 8b 15 19 77 0d 00 f7 d8 64 89 02 b8 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 07 01 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d ed 76 0d 00 f7 d8 64 89 01 48
[ 1394.006837] RSP: 002b:00007fa391ffe7e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000107
[ 1394.006840] RAX: ffffffffffffffda RBX: 00007fa37a9ce57b RCX: 00007fa3a70126fb
[ 1394.006841] RDX: 0000000000000000 RSI: 00007fa37a9ce57b RDI: 000000000000003a
[ 1394.006843] RBP: 00007fa391ffe870 R08: 0000000000000000 R09: 0000000000000001
[ 1394.006844] R10: 0000000000000003 R11: 0000000000000246 R12: 00007fa391ffe8a0
[ 1394.006846] R13: 00007fa391ffe8a0 R14: 00007fa391fff868 R15: 00007fa3940628b0
[ 1394.006851]
[ 1394.006852] ---[ end trace 0000000000000000 ]---
[ 1394.006854] BTRFS: error (device dm-0 state A) in __btrfs_unlink_inode:4144: errno=-2 No such entry
[ 1394.006858] BTRFS info (device dm-0 state EA): forced readonly
@m2Giles
Copy link
Member

m2Giles commented Jun 9, 2024

I believe in this case your btrfs array has metadata corruption. btrfs is putting the array into read-only to try and protect your data from further corruption and you should make a backup of your data asap.

After you made the backup of your data, you can try the following. Note, this is potentially destructive and usually only tried a s a last resort. You can check the btrfs wiki for possible steps to try before this.

sudo btrfs check --repair

How your system got into this state? Your ram or drive could possibly be going bad. Or you got unlucky with a bit flip during a write.

@m2Giles
Copy link
Member

m2Giles commented Jun 19, 2024

Did you also scrub the metadata?

@jholz88
Copy link
Author

jholz88 commented Jun 20, 2024 •
edited
Loading

Weird, all tests on RAM and disks passed.
Scrubbed metadata, cleared cache, no dice.
Luckily I hadn't been using Bluefin long so there wasn't much to backup.
In my experience, Bluefin was good in theory, but not in practice. It either wanted to go asleep when I needed it to stay awake, or stayed awake when I wanted it to sleep and I could never get secureboot working among other things.
So I decided to give VanillaOS Orchid a try instead and everything works out of the box. Thanks again for your help.

@m2Giles m2Giles closed this as not planned Won't fix, can't repro, duplicate, stale Jun 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@m2Giles @jholz88