Enable any ML experiment tracking framework

[williamFalcon]

People seem to have strong preferences for either using MLFlow, test-tube, polyaxon, etc...

Let's just add generic support for whatever people want to use. I don't know if generic support is possible, but each can easily be supported individually.

To make this work we'd need to:

• change the logging to be non test-tube specific. Logging only happens in 2 places (train and validation completion).
• Each call to log needs to be process-safe. Meaning when using distributed only rank=0 will log.
• the experiment param in Trainer will need to be generalized (signature the same), to take any logger.

I think that's all that's needed to add this support.

Any suggestions and takers for working on this integration?
@Borda @alok

[williamFalcon]

I also wonder if SlurmManager should just be brought into lightning. Thoughts?

[Borda]

So far I see, all used methods are following

from test_tube import HyperOptArgumentParser, Experiment, SlurmCluster

I would remove the test_tube dependence from pytorch_lightning.testing.lm_test_module so the pytorch_lightning package will be independent on these additional frameworks and then I would even keep this test_tube for testing/examples...
And instead HyperOptArgumentParser I would use standard/basic python argparse
https://docs.python.org/3/library/argparse.html

[williamFalcon]

There are also these lines (which need to be generic for each logger (but seems hard seeing they all have a different interface). So, maybe have a function that calls the appropriate thing.

here, here, here, here (basically search for self.experiment).

I kind of like the idea of limiting support for a few things here so that people aren't lost in what they can use. We can support many different ones, but would be nice in the docs if people were told which ones they can choose from.

A big help that lightning provides is helping people who don't know what they don't know. For instance some people not in CS/ML have never heard of tensorboard. But if they see it listed here, they'll know they can get the support for free if they use it.

Also remember that the experiment object in test-tube is a subclass of PyTorch SummaryWriter.

[williamFalcon]

So far I see, all used methods are following

from test_tube import HyperOptArgumentParser, Experiment, SlurmCluster

I would remove the test_tube dependence from pytorch_lightning.testing.lm_test_module so the pytorch_lightning package will be independent on these additional frameworks and then I would even keep this test_tube for testing/examples...
And instead HyperOptArgumentParser I would use standard/basic python argparse
https://docs.python.org/3/library/argparse.html

HyperargumentParser is a subclass of Argparse. It's the same thing except it adds more features for hyperparameter optimization.

[williamFalcon]

Most people also use ArgumentParser to pass in args to their models. Thus, this is the standard way of doing it in lightning. Remove the choices for things that should be standard.

A core idea that I like about Lightning is that it removes the unnecessary choices you might need to make. So it's highly opinionated about standard practices. (ie: to log we all do this, to viz we all do this, to grid search we all do this, etc...). Except instead of doing something a million ways, there's one way.

[Borda]

sure, I was thinking about having it really lightweight so even drop or move the arg parser class to this framework... about the logging, what about have one more parameter like log_callback which will be used if it is specified... and then in examples you just keep your proposal with using test-tube, but if someone come and want to use anything else, he just change the callback function... I think that Keras is also using these callbacks in a similar way...

[williamFalcon]

sure, I was thinking about having it really lightweight so even drop or move the arg parser class to this framework... about the logging, what about have one more parameter like log_callback which will be used if it is specified... and then in examples you just keep your proposal with using test-tube, but if someone come and want to use anything else, he just change the callback function... I think that Keras is also using these callbacks in a similar way...

Good suggestion, however I think that's just adding another thing for the user to remember. I'd rather them just plug in the logger and it work.

So, in the meantime, why don't we just internally call the correct function depending on the logger?

something like:

def log(logger, metrics):
    package_name = logger.__module__.split('.')[0]
   if type(logger) is 'test_tube':
       logger.ml_flow_way_of_logging(metrics)

And replace the lines of

self.experiment.log(x)

# replace with
self.log(logger, x)

Just tried this:

(ddt) comp:~ user$ python
Python 3.7.3 (default, Mar 27 2019, 16:54:48)
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from test_tube import Experiment
WARNING:root:This caffe2 python run does not have GPU support. Will run in CPU only mode.
>>> Experiment.__module__
'test_tube.log'
>>>

[karanchahal]

I'd like to pick one of the frameworks up, MLflow if it's isn't already taken. I'll look at it tomorrow morning and try to figure out how it can be integrated. I'm a bit confused though, can't tensorboard be used to track experiments and different model runs?

Also, could lighting add a feature for quantisation or pruning ?

Maybe expose an API so that people can try out different quantisation and pruning strategies to compress their Network.

I recently was able to quantise a fp32 model to 8 bits ( only the fully connected and conv layers) with next to no loss in accuracy, this was an MNIST network. But I'm following and using the same algos the Tensorflow lite people are using. I can share a Google colab notebook of a sample implementation of you think it's a viable thing for this project.

[williamFalcon]

@karanchahal I agree with you... most people I know use tensorboard, but happy to allow this to be flexible. There's no real need to force an opinion on this particular thing from the framework.

I think @Borda is probably going to look at it (but we can defonflict here if you want to look at it).

However, a way of doing network pruning and quantisation would be good. I don't know those in details, so maybe make a proposal and what the use cases are for? (different issue with enhancement tag)

[Borda]

I would keep #44 simple just adding CI, so I would leave the resolving test-tube to another PR/user 😄

[williamFalcon]

@karanchahal feel free to give this a shot!
link to this issue