Segfault when running KL Divergence on both Windows & WSL

[kalomaze]

I followed the instructions outlined in the KL Divergence PR to the best of my ability:

• perplexity.exe -m "C:\Users\Kalo\Documents\GitHub\llamacpp_git\examples\dpo_7b_quant\DPO_7b_q8_0.gguf" -f "C:\Users\Kalo\Downloads\8k_redo.txt" --kl-divergence-base 8k_redo_data.dat -ngl 33

On disk, the file is saved:

Then, I use this data file like how it was specified to be used in the PR, and it silently errors:

• perplexity.exe -m "C:\Users\Kalo\Documents\GitHub\llamacpp_git\examples\dpo_7b_quant\DPO_7b_IQ2_XXS_combrandt1.gguf" --kl-divergence-base "C:\Users\Kalo\Documents\GitHub\llamacpp_git\cmake_build\bin\Release\8k_redo_data.dat" --kl-divergence -ngl 33

Attempting to repeat this process from start to finish on WSL rather than Windows caused a segfault after it hanging for ~20 seconds.
Attempting to remove kl-divergence mentions that the data "only tokenizes to one token".

perplexity: saving all logits to C:\Users\Kalo\Documents\GitHub\llamacpp_git\cmake_build\bin\Release\8k_redo_data.dat
perplexity: tokenizing the input ..
perplexity: tokenization took 0.459 ms
perplexity: you need at least 1024 tokens to evaluate perplexity with a context of 512
perplexity: the data file you provided tokenizes to only 1 tokens

I tried cleaning the build files in Visual Studio, reconfiguring cmake, rebuilding from scratch, etc, and I get the same issue.
Perplexity and imatrix calculations otherwise work as you would expect, I have no issues with those.

[kalomaze]

@ikawrakow Sorry to bother, but I haven't been able to get this working at all.
I'm going to try to completely wipe cmake cache and see if that helps.

EDIT: I'm able to get the file generated again as .dat. But I get the same issue of it silently crashing.

[kalomaze]

It appears to be related to some sort of permissions issue. That seems very odd to me that this could cause a segfault. I think I can fix the issue with elevated permissions in WSL. (Either that, or fresh WSL reinstall fixed it)

[kalomaze]

The same exact file is reporting a non-margin of error difference in perplexity on Windows. So it's presumably not tokenizing the same way as it does on WSL, at least with the KL divergence stuff.

Worrying, because if it happens outside of KL div calculations / general perplexity calculations... that implies something is really wrong, I think.

On WSL:
./perplexity -m "DPO_7b_q8_0.gguf" -f "8k_redo.txt" --kl-divergence-base 8k_redo_data.dat -ngl 33
Final estimate: PPL = 10.3204 +/- 0.42011

On Windows:
perplexity.exe -m "C:\Users\Kalo\Documents\GitHub\llamacpp_git\examples\dpo_7b_quant\DPO_7b_q8_0.gguf" -f "C:\Users\Kalo\Downloads\8k_redo.txt" --kl-divergence-base 8k_redo_data.dat -ngl 33
Final estimate: PPL = 11.1456 +/- 0.46025

Tried git cloning from source again in a new folder, and resetting CMake, and this doesn't change:

Final estimate: PPL = 11.1460 +/- 0.46027

[kalomaze]

Tried setting the context for PPL evaluation on this 8k tokens file to 128.

WSL: 13.4018 +/- 0.59528
Windows: 13.9301 +/- 0.62122

The difference is still noticeable... that's really concerning. I'm on q8_0.

EDIT: Ok, so it looks like it may have to do with how the input file gets tokenized. Perhaps a Unix vs Windows line endings thing?

ON WSL:

perplexity: calculating perplexity over 307 chunks, batch_size=32

ON WINDOWS:

perplexity: calculating perplexity over 294 chunks, batch_size=32

Also, attempting to actually use the generated .dat file that was made for KL divergence purposes (on Windows) causes the file to be completely erased and replaced with this:

It just doesn't make sense to me why it would try to write to the file. Does anyone have any clue at all what could be going on here or how I can debug?

[kalomaze]

I believe I have identified the issue, and made a PR that should fix it.
Should be investigated on whether or not this change should be done for the imatrix example as well.

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.