`quantize`: add imatrix and dataset metadata in GGUF

[phymbert]

Motivation

I was reading thanks to @julien-c this reddit post from @he29-net 👍

You can't easily tell whether a model was quantized with the help of importance matrix just from the name. I first found this annoying, because it was not clear if and how the calibration dataset affects performance of the model in other than just positive ways. But recent tests in llama.cpp discussion #5263 show, that while the data used to prepare the imatrix slightly affect how it performs in (un)related languages or specializations, any dataset will perform better than a "vanilla" quantization with no imatrix. So now, instead, I find it annoying because sometimes the only way to be sure I'm using the better imatrix version is to re-quantize the model myself.

Proposal

• Add at the end of the imatrix binary file the dataset name on which the imatrix was computed on

• Add following KV in quantize:

  • quantize.imatrix.file Filename of the provided imatrix during quantization
  • quantize.imatrix.entries_count Number of entries in the imatrix
  • quantize.imatrix.dataset Dataset from the imatrix
  • quantize.imatrix.chunks_count Number of chunks the imatrix was computed with

Ideally I would also add both imatrix and dataset files hashes in the metadata, but I am not sure this is supported and appropriate.

[phymbert]

@ggerganov @ikawrakow Thoughts ? Can I give it a try ?

[sorasoras]

What happen if you combine different imatrix? if you combine a lots of .dat, how would metadata work?

[phymbert]

What happen if you combine different imatrix? if you combine a lots of .dat, how would metadata work?

As far as I understand, It will add the metadata from the final imatrix, but you are pleased to test.