Parameters for fine-tuning BB3 3B with additional task

[krisstud]

I'm looking to fine-tune the BB3 3B model with an additional (custom) fine-tuning task. The example parameters for fine-tuning, as far as i understand them, entails fine-tuning using all the datasets from the base R2C2 model and so on. Now, I'm just looking to fine-tune from the finished BB3 model with one more dataset. Are these params correct for that task? :

task='projects.bb3.tasks.r2c2_dialogue_tasks:CustomVanillaDialogueTeacher', # customized py file with my task multitask_weights='1' # no multi-task training - at least for now model='projects.seeker.agents.seeker:ComboFidGoldDocumentAgent', # to not do search during training? init_model='zoo:bb3/bb3_3B/model', init_opt='arch/r2c2_base_3B'

The rest are identical to the example for fine-tuning:
-vstep 1000 -lstep 50 --batchsize 1 \ --model projects.seeker.agents.seeker:ComboFidGoldDocumentAgent \ --n-docs 5 --text-truncate 1024 --label-truncate 128 --truncate 1024 --fp16 True \ -lr 1e-06 --lr-scheduler reduceonplateau --lr-scheduler-patience 3 --optimizer adamw \ --save-after-valid True --warmup-updates 100 --update-freq 1 --gradient-clip 1.0 --skip-generation True \ --dropout 0.1 --attention-dropout 0.0 -vp 10 -vmt ppl -vmm min -vme 100000 --load-from-checkpoint true \ --ddp-backend zero2 --checkpoint-activations true --model-file /path/to/model/

The doc does not say, but i assume we use the 'gen' opts for inference only, and 'arch' opts for training.

Edit:
Or maybe the correct model parameter would be:
model='projects.bb3.agents.r2c2_bb3_agent:ComboFidGoldDocumentAgent', or
model='projects.bb3.agents.r2c2_bb3_agent:BB3SubGoldAgent',

[klshuster]

your setup looks like it would work!

[krisstud]

If people come looking, this is the SLURM command i am going to use (starts training correctly, i think) used.
It does report | Total parameters: 673,069,056 (673,069,056 trainable), which seems low, but the log reports using the BB3 3B init_model and agent, and perplexity scores do seem in line with that. The 4 nodes are 80GB A100.

srun python -u -m parlai.scripts.distributed_train \
    --task projects.bb3.tasks.r2c2_dialogue_tasks:custom_datasetVanillaDialogueTeacher,projects.bb3.tasks.r2c2_dialogue_tasks:LightWildVanillaDialogueTeacher --dict-file cluster/home/username/code/ParlAI/data/custom_dataset/premade.dict,cluster/home/username/code/ParlAI/data/light_dialogue_wild/premade.dict \
    --multitask-weights 2,0.5 \
    --evaltask projects.bb3.tasks.r2c2_dialogue_tasks:custom_datasetVanillaDialogueTeacher,projects.bb3.tasks.r2c2_dialogue_tasks:LightWildVanillaDialogueTeacher \
    --vstep 1000 -lstep 50 --batchsize 32 --init-opt 'arch/r2c2_base_3B' \
    --init-model 'zoo:bb3/bb3_3B/model' \
    --model projects.seeker.agents.seeker:ComboFidGoldDocumentAgent \
    --n-docs 5 --text-truncate 1024 --label-truncate 768 --truncate 1024 \
    --fp16 true --fp16-impl safe -lr 1e-06 --lr-scheduler reduceonplateau --lr-scheduler-patience 3 \
    --embedding-size 2048 --ffn-size 8192 --optimizer adamw \
    --save-after-valid True --warmup-updates 100 --update-freq 1 --gradient-clip 1.0 --skip-generation True \
    --dynamic-batching none \
    --dropout 0.1 --attention-dropout 0.0 -vp 10 -vmt ppl -vmm min -vme 100000 \
    --ddp-backend zero2 --checkpoint-activations true --model-file /cluster/work/username/models/bb3_split/bb3_split \
    --distributed-world-size 4 \
    --wandb-log true --wandb-project projectname \

[klshuster]

The param count printed is the total parameters divided by the number of GPUs, so 4 * 673 ~= 2.7B