Closed
Description
See: https://twitter.com/ggerganov/status/1640945226662420483
The gpt4all ggml model has an extra <pad>
token (i.e. n_vocab = 32001
).
Need to add it during the conversion. Should be an optional command line argument to the script to specify if the token should be added or not