MPT : clone wte to output at load time

[cebtenzzre]

This change removes the need to duplicate the token_embd.weight tensor on disk. Instead, it is cloned as output.weight at load time.

The tensor is not cloned while quantizing because quantize does not call llm_load_tensors.

[slaren]

I don't think that it is a good idea to add what is essentially a hack to gguf to deal with a llama.cpp loader shortcoming. If the tensor really needs to be duplicated (and to be fair, llama.cpp may need to in some cases for partial offloading with CUDA), then that should be handled in the application. The special case for MPT in llm_load_tensors is not great either, it will make harder to refactor this code in the future.

There may also be an unintended side-effect of this, which is that it will prevent using different quantizations for the token embeddings and the output layer. From what I understand, the quantization of the output tensor has a large impact on generation quality, while the token embeddings has very little impact. For that reason the output tensor is usually quantized at a higher bit rate.

[apage43]

different quantizations for the token embeddings and the output layer

for models where they're the same data it seems fine to just use the highest you'd choose for either - that is, look up the token embeddings in higher precision for models that do this

Mostly I find the file-level hacks more regrettable than code level ones, like before MQA/GQA were well supported some implementations of models that used them would duplicate the the data that would be broadcasted in the file which not only bloated the file unnecessarily but meant the file format was going to change again later if those limitations were dealt with.

really needs to be duplicated (and to be fair, llama.cpp may need to in some cases for partial offloading with CUDA),

maybe this is where this should actually be solved? - this is not the only conceivable case where the same tensor may be used more than once in a single model/graph

the model code looking like a model that doesn't do tying is potentially confusing for anyone comparing it against the original code that doesn't know about this workaround, and as I understand it writing it like the original does work when not offloading

[cebtenzzre]

@slaren Is there any better way you think we can accomplish this, or should I close this PR and convert the MPT-based GPT4All models to save the output tensor twice in the GGUF file?

[slaren]

The quantization issue probably could be solved by exporting the tensor as the output layer rather than as the token embeddings, then the higher quality quantization would be automatically chosen. Ideally, we would also want to keep only one copy of the tensor in memory, but there is no escaping the fact that with partial offloading to GPU, sometimes we would also need a copy of the tensor in VRAM. So we would need to check if the backend of the token embeddings and the output layer is different, and then copy the tensor as required.

However, supporting this cleanly in the llama model loader code in a way that isn't just a hack for this specific case would require significant changes. I think this may become somewhat easier to do in the future with ggml-backend, since there will be a common interface for copying tensors between different backends, but as it is now, I don't think it is worth adding all this code to llama.cpp just to save a few MB in the model file. That's just my opinion, though.