Adding support for persistent workers and logger fix #198
Conversation
|
Thanks for this! Adding @paulmorio for review! We'll review asap 🙂 |
| @@ -60,6 +61,10 @@ def init_trainer(self) -> Tuple[Trainer, ModelCheckpoint]: | |||
| config = self.trainer_config | |||
| config = _add_pyl_trainer_defaults(config) | |||
| callbacks: List[Callback] = [] | |||
|
|
|||
| if config["logger"] is None: | |||
There was a problem hiding this comment.
I think default should remain with no logger, at the very least there should be an option to not have a logger.
There was a problem hiding this comment.
Hi, Thomas. I was trying to match the Pytorch Lightning API which sets as None and uses default behavior of CSVLogger. To have no logger - consistent with the lightning API - you'd set it as logger=False. So we currently support all this functionality and we match the defaults of PyTorch Lightning. If you want to change it however - we can!
There was a problem hiding this comment.
I see, but wouldn't that mean that we don't need this since by default None would get converted as CSVLogger anyway by the Trainer?
What is the goal of this PR?
This PR adds enhanced configurability and stability to data loading and logging features within pyrelational's DataManager and LightningModelManager classes. These changes aim to improve both the efficiency of data handling and logging management by adding support for persistent workers in data loading and establishing the ability to pass loggers to the pytorch lightning trainer. This update ensures that data loading can better utilize resources over prolonged tasks and provides support for pytorch lightning loggers.
What are the changes implemented in this PR?
Persistent Workers in DataManager:
The
DataManagerclass now includes an option to keep workers persistent across data loading iterations, controlled by a newloader_persistent_workersparameter. By settingloader_persistent_workers: bool = Falseby default, backwards compatibility is maintained, but it can be enabled to reduce initialization overhead in multi-epoch training, benefiting users handling larger datasets or needing faster data loader setup times.In the
LightningModelManagerclass, default logging has been configured using PyTorch Lightning's CSVLogger. If no logger is specified in trainer_config, a CSVLogger is automatically set up to log to config["checkpoints_dir"] consistent with pytorch lightning.Lastly, the abstract model manager now uses a context manager to load JSON data.