config.json for 4bit quantized files

[philwee]

I've been able to convert files from HF format to f16 and 4bit, but I've not been able to figure out what config.json (or what changes to the config.json) to use when attempting to evaluate 4 bit quantized models. (Which is needed when trying to evaluate models)

I've tried to just change the torch_dtype to int4, but that doesn't seem to work.

For context, this is that the current config.json for llama 13b looks like:

{
  "architectures": [
    "LlamaForCausalLM"
  ],
  "bos_token_id": 1,
  "eos_token_id": 2,
  "hidden_act": "silu",
  "hidden_size": 5120,
  "initializer_range": 0.02,
  "intermediate_size": 13824,
  "max_position_embeddings": 2048,
  "max_sequence_length": 2048,
  "model_type": "llama",
  "num_attention_heads": 40,
  "num_hidden_layers": 40,
  "pad_token_id": 0,
  "rms_norm_eps": 1e-06,
  "tie_word_embeddings": false,
  "torch_dtype": "float16",
  "transformers_version": "4.28.0.dev0",
  "use_cache": true,
  "vocab_size": 32000
}

I would really appreciate any help. Thank you!

[philwee]

Update: turns out lm evals does not support 4 bit quantizes yet, will need to work on that first (or if anyone wants to work on it, please see: EleutherAI/lm-evaluation-harness#417

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.