Lost writes after 5M unique keys are inserted in two partions.

[uniphil]

Describe the bug

Fjall appears to sometimes lose writes after closing and reopening a keyspace with two partitions.

To Reproduce

My repro is here: https://github.com/uniphil/kv-for-likes-test/blob/fjall-lost-write-repro/fjall/src/lost-write.rs
With data file here: https://github.com/uniphil/kv-for-likes-test/blob/fjall-lost-write-repro/likes5M-anon.txt

It has 5M entries total between the two partitions -- at 4M it wasn't reliably reproducing.

• Run with cargo run --release --bin lost-write

• It fails every time for me with an output like

  olde-mbp:fjall phil$ cargo run --release --bin lost-write
     Compiling kv-for-likes_fjall v0.1.0 (/Users/phil/code/kv-for-likes/fjall)
      Finished `release` profile [optimized] target(s) in 2.06s
       Running `target/release/lost-write`
  done in 24.8s. likes: 4932348, unlikes: 67652
  checking likes: OK:   4932348 == 4932348
  checking unlikes: OK:   67652 == 67652
  reopening keyspace...
  checking likes: OK:   4932348 == 4932348
  checking unlikes: FAIL: 67004 != 67652
  

with this size of data file, it's always the unlikes count that's wrong after reopening. with the full 17.5M set i think both were wrong (but i'd need to confirm). The number of lost writes to unlikes varies across runs.

I wasn't seeing this issue before when I had set a larger block size.

Expected behavior

The number of keys in a partition shouldn't change after closing and reopening it.

Screenshots

n/a

Desktop (please complete the following information):

• OS: macOS 10.15.7 (yeah i know)
• cargo/rust: 1.82.0
• fjall: 2.4.1

[uniphil]

confirmed the above repro also on macos 14.7.1 / rust 1.81, so it's not only due to my machine being old and sad.

edit: also confirmed with fjall 2.4.2 on the old machine.

edit 2: also confirmed repro on my next-most-convenient machine... a raspberry pi 4b (linux raspberrypi) on an XFS volume. (mac machines were APFS). so likely not FS, OS, or architecture specific.

[marvin-j97]

@uniphil I think I tracked it down to the journal truncation, which is kind of too convoluted anyway.

Can you try this branch? https://github.com/fjall-rs/fjall/tree/fix/journal-truncation
That works for me, though I don't fully understand where there previous implementation failed.

[uniphil]

yup it worked! 🎉

olde-mbp:fjall phil$ cargo run --release --bin lost-write
    Updating git repository `https://github.com/fjall-rs/fjall.git`
     Locking 1 package to latest compatible version
      Adding fjall v2.4.2 (https://github.com/fjall-rs/fjall.git?branch=fix/journal-truncation#4f042af8)
   Compiling fjall v2.4.2 (https://github.com/fjall-rs/fjall.git?branch=fix/journal-truncation#4f042af8)
   Compiling kv-for-likes_fjall v0.1.0 (/Users/phil/code/kv-for-likes/fjall)
    Finished `release` profile [optimized] target(s) in 9.09s
     Running `target/release/lost-write`
done in 23.3s. likes: 4932348, unlikes: 67652
checking likes: OK:   4932348 == 4932348
checking unlikes: OK:   67652 == 67652
reopening keyspace...
checking likes: OK:   4932348 == 4932348
checking unlikes: OK:   67652 == 67652

[marvin-j97]

https://crates.io/crates/fjall/2.4.3