assertion failed: datablocks_to_sectors(upper_data_aligned) + upper_limit_meta_size * 2u64 > total_space

[refi64]

I just upgraded to Stratis 3.2.1 yesterday, and while working with some files today (basic operations, just updating the cover art on some music files), all I/O completely froze & I got this error in the journal:

Aug 17 09:45:54 hp-fedora stratisd[1732]: thread 'stratis-wt-657' panicked at 'assertion failed: datablocks_to_sectors(upper_data_aligned) + upper_limit_meta_size * 2u64 >\n    total_space', src/engine/strat_engine/thinpool/thinpool.rs:303:5
Aug 17 09:45:54 hp-fedora stratisd[1732]: stack backtrace:
Aug 17 09:45:54 hp-fedora stratisd[1732]: note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Aug 17 09:45:54 hp-fedora stratisd[1732]: [2022-08-17T14:45:54Z WARN  stratisd::engine::strat_engine::engine] Failed to get status for thread handling filesystem checks: task 1301 panicked

I'm...admittedly not sure how to debug this, I'm guessing there's some FS metadata or similar that would help?

[mulkieran]

@refi64 Thanks for the report! We believe that we understand the cause for the assertion failure, and can provide a fix. We'll provide further updates as we work out the details.

[jbaublitz]

@refi64 I have a questions to try to narrow this down a bit. Do you have an idea of whether the thin pool was mostly allocated? (As in AllocatedSize D-Bus property was very close to TotalPhysicalSize) Are you able to check or does it panic each time you try to start up stratisd? Would you be willing to test a fix that I believe should resolve the issue if it's what I think it is?

[refi64]

Do you have an idea of whether the thin pool was mostly allocated? (As in AllocatedSize D-Bus property was very close to TotalPhysicalSize)

It's fully allocated (apparently I like to live dangerously):

$ busctl get-property org.storage.stratis3 /org/storage/stratis3/0 org.storage.stratis3.pool.r2 AllocatedSize
s "1994702979072"
$ busctl get-property org.storage.stratis3 /org/storage/stratis3/0 org.storage.stratis3.pool.r2 TotalPhysicalSize
s "1994702979072"

Are you able to check or does it panic each time you try to start up stratisd?

stratisd stays running the whole time, and I seem to be getting the panics every ~10 seconds.

[mulkieran]

@refi64 Would you like to try out #3086 #3096? We believe that we have identified the exact problem that was causing the panic, and that this PR should fix it.