Experiment tracking support with ClearML Logger v1

[skinan]

Patch description

I have added ClearML Logger to log experiments using ClearML Task. ClearML logger logs everything to ClearML WebUI. ClearML logs Hyperparameter, Scalars, Models, Checkpoints, other necessary artifacts such as dictionaries, and also Test and Validation Dialogue Worlds as debugging samples.

• Added ClearMLLogger Class (Similar to TensorboardLogger and WandbLogger) to parlai/core/logs.py
• Additional code added to files under parlai/scripts/. These scripts are: train_model.py .
• No other scripts or files are changed in terms of adding codes.

ClearML is a leading MLOps stack that can supercharge dialogues research with its state-of-the-art experiment tracking capability. ClearML: https://clear.ml/

What ClearML Experiment Manager can log? Everything! You just name it. Example Screenshots:

Scripts for testing ClearML Logger

test_clearml_scripts.zip

How to install ClearML

https://clear.ml/docs/latest/docs/getting_started/ds/ds_first_steps/

[klshuster]

Hi there, thank you for your contribution. We appreciate your interest in logging within ParlAI, though I have a few questions:

1. Do you have more signal on whether other folks would find this useful in ParlAI? We currently offer full support for using tensorboard and weights & biases, which seem to cover most if not all of what is offered in this solution.
2. If we decide to include this in ParlAI, could you please add unit tests to via mocking to ensure long-standing correctness?

[skinan]

Hi @klshuster, thanks for the feedback!

1. We've heard from several users that they would appreciate the convenience that the integration of parlai and ClearML will bring. While we don't have any quantitative answer, we've heard from a few folks that use both parlai and ClearML that they would consider using an integration.

2. As for the differences between ClearML and Tensorboard \ Wandb, ClearML offers similar tracking features to Wandb but is completely open source (incl. the server) so there's a difference in audience. Additionally, ClearML has an orchestration feature that makes it easy to run parlai jobs on other machines, which we know is a compelling feature for many users. As for Tensorboard, the main difference is that ClearML (Like Wandb) has an experiment tracking system that logs everything, not just metrics, which allows users to go back in time and rebuild their code, compare experiments, download any model, and more.

We believe that this integration brings a lot of value, and we'd be happy to add tests if you agree.

Thanks a lot!

[klshuster]

Ok, please add some tests and then I can review the full PR

[github-actions]

This PR has not had activity in 30 days. Closing due to staleness.

[skinan]

Working on the tests.

[skinan]

Ok, please add some tests and then I can review the full PR

@klshuster , I have added unittest. Please, give a review to the PR.

[skinan]

@mojtaba-komeili , request for a review.

[mojtaba-komeili]

Thanks for contributing to ParlAI.