Vighnesh Subramaniam, Yilun Du, Joshua B Tenenbaum, Antonio Torralba, Shuang Li, Igor Mordatch
This is the implementation of our paper "Multiagent Finetuning of Language Models". We design this implementation for the MATH dataset for simplicity. Using other datasets requires little modification of the code.
We include a requirements.txt to provide the basic requirements for set-up. To do finetuning with open-source language models, we also include a more details set up in the path multiagent-ft/lm_ft with a more detailed list of required packages.
Set your OpenAI API Key using export OPENAI_API_KEY=your_api_key_here.
Create a conda/pip environment and install Pytorch. Then run
pip install -r requirements.txt
To download the MATH dataset, follow the steps here.
We ran all experiments either with GPT-3.5 with the OpenAI API or with open-source language models on four 40GB A100s or 8 H100s.
To create finetuning data, change directory to the multiagent-ft directory and run
python original_gen.py --agents [AGENTS] --rounds [ROUNDS] --model [MODEL] --save_str [SAVE_STR] --summarize
where agents refers to the number of multiagent debate agents, rounds refers to the number of rounds of debate, model refers to the model to use debate on, and save_str refers to the save log. summarize is an argument that makes the model summarize responses from other agents. We also include parameters for top_p and temperature that can be used with the open-source models.
To create data for finetuning generation agents, run
python ft_generator.py --file_path [FILE_PATH] --save_path [SAVE_PATH] [--gpt] --iteration [ITERATION]
This takes in a path for a JSON file and creates the JSON/JSONL files for finetuning. If you include the --gpt flag, it will use the OpenAI API for finetuning assuming the goal is to finetune GPT-3.5. You can track which iteration of finetuning you're applying using the --iteration flag. You also can pass in the GPT model IDs to finetuning using the model_ids flag.
Similar to generator finetuning. Run
python ft_critic.py --file_path [FILE_PATH] --save_path [SAVE_PATH] [--gpt] --iteration [ITERATION]
To finetune open-source models, refer to the lm_ft directory.
To run finetune models, pass either the model ID strings for OpenAI models or the model paths for the finetuned models as arguments when running
python ft_gen.py --generators [GENERATORS] --critics [CRITICS] --model [MODEL] --save_str [SAVE_STR]
To account for hardware/memory limitations, this will put all open-source models from HuggingFace on the same GPU.
To evaluate the performance, set the correct path for the JSON file and run
python eval_math.py