[train] New training options for logging/validation based on number of steps

[klshuster]

Are you here because you got a warning?

You have been warned for setting one of the following options during a distributed training run (multiprocessing_train or distributed_train):

• Stopping criteria: --num-epochs/-eps, --max-train-time/-ttim
• Logging criteria: --log-every-n-secs/-ltim
• Validation criteria: --validation-every-n-sec/-vtim, --validation-every-n-epochs/-veps

In order to have any of these options work in distributed training, we must perform a status synchronization between all workers on every single training batch. This is a significant communication cost, and slows down training. You may still use these options, but your training could be faster.

Instead, if you limit yourself to these options, you may see as much as a 5-15% speedup in your training.

• Stopping criteria: --max-train-steps/-tstep (too dependent on data/model to recommend values)
• Logging criteria: --log-every-n-steps/-lstep (good values are 10, 50, or 100)
• Validation criteria: --validation-every-n-steps/-vstep (good values are 100, 500, or 1000)

---

Patch description

To add to the litany of options for controlling logging, validation, and training, we now have 3 brand new, shiny options:

• -lstep, --log-every-n-steps - log every n training updates
• -vstep, --validation-every-n-steps - run validation every n training updates
• -tstep, --max-train-steps - train for no more than n training updates

As a result, --max-lr-steps is now deprecated (It's still respected, but discouraged). Additionally, --max-lr-steps now applies to ALL LEARNING RATE STEPS; prior to this change, the option only applied to steps post-warmup.

Speed test

Testing a 1000 step training run on a 287M parameter transformer/generator with the GPT2 tokenizer, with no validations, and syncing turned on and off.

Testing steps

Manual testing and new CI. Manual benchmarking

Correctness

Added new tests to test_train_model.py

$ pytest -k TestTrainModel
.
.
.
tests/test_train_model.py ...... [100%]

==== slowest 10 durations ====
5.85s call     tests/test_train_model.py::TestTrainModel::test_opt_step
5.06s call     tests/test_train_model.py::TestTrainModel::test_opt_step_update_freq_2
1.74s call     tests/test_train_model.py::TestTrainModel::test_multitasking_metrics_micro
1.71s call     tests/test_train_model.py::TestTrainModel::test_multitasking_metrics_macro
0.30s call     tests/test_train_model.py::TestTrainModel::test_fast_final_eval
0.01s call     tests/test_train_model.py::TestTrainModel::test_multitasking_id_overlap

(4 durations < 0.005s hidden.  Use -vv to show these durations.)
==== 6 passed, 865 deselected, 2 warnings in 20.49s ====

[stephenroller]

I still think we need to remove the metadata syncs at the top of the while True loop, if we're only using these new options.

[stephenroller]

Let's bump this to an error. We already have enough warns that people ignore

[klshuster]

i think you're right, as my distributed sweeps are failing atm

[emilydinan]

lgtm!

[github-actions]

This PR has not had activity in 30 days. Closing due to staleness.

[klshuster]

i've just started a sweep, hopefully will have results soon

[stephenroller]

I reviewed and can't see a reason why it would have the inverse effect we predicted. Can we rerun our benchmark?

[stephenroller]

Maybe your benchmark has a max train time? Or maybe our sweeper adds one?

[stephenroller]

We've validated the speedups from this change in offline tests.