Feat/tensorboard integration

[nick-rfai]

Changes

• Add new start script colab action
• Integrate tensorboard into new metric_logger ABC
• Create a new interactive controller python widget for interacting with dispatcher during colab
• Allow run_fit to be backgrounded so that users can access python widget during training

Testing

• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I have tested this change manually
• I have tested this change in the following environments:
  • Local development
  • Colab
  • PBD
  • Regression tests pass on the lite notebooks

Screenshots + Test Steps

1. For Colab notebook, you need a T4 GPU and you may need to purchase additional credits if you run multiple experiments. Also, the notebook auto shuts down after 5-10 minutes of no activity.
2. pip install git+https://github.com/RapidFireAI/rapidfireai.git@feat/tensorboard-integration, ignore pip warnings.
3. !rapidfireai init in a cell to download the tutorial notebooks, and bring over cells from colab-tensorboard notebook.
4. Open Colab terminal, export tensorboard param + run start script

export RF_TRACKING_BACKEND=tensorboard
rapidfireai start --colab

5. Run in Colab cell

import os
os.environ['RF_TRACKING_BACKEND'] = 'tensorboard'

%reload_ext tensorboard

6. Run GPT2 experiment
7. Make sure tensorboard widget shows up

8. Make sure IC widget shows up (there should be no data here until run_fit is active)
9. Run the run_fit cell. Note that on colab free tier account, there is an automatic timeout which will shutdown the notebook.
10. Perform clone-modify or other command in IC op widget

Regression Test

Checklist

• My code follows the project's style guidelines
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published

TODO

• Verify new test cases work
• Remove use of mlflow for colab + tensorboard version (can we also remove from install script?)
• Run gpt2 notebook e2e
• Update notebook to match colab test notebook
• Make sure github version of notebook is syntax error-free
• Test that no regressions introduced, shutdown script still works with non-colab path

Performance Impact

If this PR affects performance, describe the impact and any optimizations made.

Related Issues

Fixes #(issue number)
Closes #(issue number)
Related to #(issue number)

[arun-rfai]

The steps and the end to end demo look good to me. The gpt-2 notebook also looks good. Wait for PS to finish review before merging.