Skip to content

Latest commit

 

History

History
45 lines (27 loc) · 2.04 KB

advanced-usage.md

File metadata and controls

45 lines (27 loc) · 2.04 KB

Advanced Usage

Note that the following sections describe how to use the fine-tuning CLI for advanced purposes. To do this on Colab, simply use the arguments mentioned here in the argvec list in our Colab notebook

Using any Huggingface Model

python3 -m fine_tune.cli --model <HF name*> --dataset <dataset name> --lang <iso lang code> --iglue_dir <base path to indic glue dir> --output_dir <output dir>

where HF name refers to the Huggingface shortcut name for the model. For the list of all shortcut names, refer the official docs https://huggingface.co/transformers/pretrained_models.html

Loading Model from Local File

All models in the code are loaded through HF transformers library. For any model, you need the following three files:

  • config.json: config file in HF format; check config files used by transformers, for example here.
  • tok.model: the tokenizer (spm, wordpiece etc.) model file.
  • pytorch_model.bin: pytorch binary of the transformer model which stores parameters.

If you have tensorflow checkpoints instead of pytorch binary, then use the following command to first generate the pytorch binary file:

MODEL_DIR=$1

# modify model_type and filenames accordingly
transformers-cli convert --model_type albert \
  --tf_checkpoint $MODEL_DIR/tf_model \
  --config $MODEL_DIR/config.json \
  --pytorch_dump_output $MODEL_DIR/pytorch_model.bin

Finally, run the evaluation using the following command:

python3 -m fine_tune.cli --model <path to the directory containing pytorch_model.bin> --tokenizer_name <path to the tokenizer file> --config_name <path to the config file> --dataset <dataset name> --lang <iso lang code> --iglue_dir <base path to indic glue dir> --output_dir <output dir>

Running Cross-lingual Experiments

Add later