feat: help smooth out losses' value on resume

[SamuelLarkin]

PR Goal?

This PR fixes the weird spikes in tensorboard when resuming training.

Fixes?

fixes #419

Feedback sought?

Review to merge the solution.

Priority?

This is part of alpha so may be medium-high

Tests added?

No

How to test?

for max_epochs in 3 6 9 12 15; do
   echo "Training up to $max_epochs"
   srun everyvoice train text-to-spec \
      config/everyvoice-text-to-spec.yaml \
      --config-args training.finetune_checkpoint=$PWD/logs_and_checkpoints/FeaturePredictionExperiment/base/checkpoints/last.ckpt \
      --config-args training.max_epochs=$max_epochs
done

then

tensorboard --port=2024 --logdir=logs_and_checkpoints  --bind_all

Confidence?

Good

Version change?

No

Related PRs?

EveryVoiceTTS/FastSpeech2_lightning#82

Tensorboard

Here are the losses graphs after 4 resume.
The curves are much smoother.

With the Fix

Prior to the Fix

[github-actions]

CLI load time: 0:00.23
Pull Request HEAD: f652073cd437169e59811cb604c0be83e91475ea
Imports that take more than 0.1 s:
import time: self [us] | cumulative | imported package

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 10 lines in your changes missing coverage. Please review.

Project coverage is 74.30%. Comparing base (a242650) to head (f652073).

Files	Patch %	Lines
everyvoice/base_cli/callback.py	0.00%	9 Missing ⚠️
everyvoice/base_cli/helpers.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #473      +/-   ##
==========================================
- Coverage   74.56%   74.30%   -0.27%     
==========================================
  Files          44       45       +1     
  Lines        2827     2837      +10     
  Branches      435      435              
==========================================
  Hits         2108     2108              
- Misses        632      642      +10     
  Partials       87       87              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[roedoejet]

Tested it out with some other data and it works as expected - this is fantastic @SamuelLarkin - thanks for all your hard work on this. I much prefer the logs like this 😎