Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make default tqdm dict overridable #749

Merged
merged 4 commits into from
Feb 5, 2020
Merged

Make default tqdm dict overridable #749

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
57094c8
overridable tqdm_dict
kuynzereb Jan 25, 2020
fef3ada
Slim down default tqdm_metrics
kuynzereb Jan 25, 2020
7a7077a
gpu fix
kuynzereb Jan 26, 2020
174b409
Merge branch 'master' into make_tqdm_dict_overridable
kuynzereb Jan 26, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions pytorch_lightning/core/lightning.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1211,6 +1211,25 @@ def on_save_checkpoint(self, checkpoint):
"""
pass

def get_tqdm_dict(self):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make it as property like a tqdm_params?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, at first I made it property. But then I accidentally found out that it will cause misleading error messages, if there is an attribute error in the user code. What I mean, if you make

@property
def tqdm_dict(self):
    _ = self.trainer.some_non_existing_attribute
    return {}

Then after

return dict(**ref_model.tqdm_dict, **self.tqdm_metrics)

you will get

AttributeError: 'CoolSystem' object has no attribute 'tqdm_dict'

I don't know how to fix it, but it seems to be the known issue: https://stackoverflow.com/questions/15348331/why-does-property-decorator-show-object-has-no-attribute

Overall, I decided to stay without property, so the user gets the clear error message if there is some error in his code.

r"""
Additional items to be displayed in the progress bar.

Return:
Dictionary with the items to be displayed in the progress bar.
"""
tqdm_dict = {
'loss': '{:.3f}'.format(self.trainer.avg_loss)
}

if self.trainer.truncated_bptt_steps is not None:
tqdm_dict['split_idx'] = self.trainer.split_idx

if self.trainer.logger is not None and self.trainer.logger.version is not None:
tqdm_dict['v_num'] = self.trainer.logger.version

return tqdm_dict


def load_hparams_from_tags_csv(tags_csv):
if not os.path.isfile(tags_csv):
Expand Down
5 changes: 2 additions & 3 deletions pytorch_lightning/trainer/evaluation_loop.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -295,7 +295,7 @@ def run_evaluation(self, test=False):
desc = 'Testing' if test else 'Validating'
pbar = tqdm(desc=desc, total=max_batches, leave=test, position=position,
disable=not self.show_progress_bar, dynamic_ncols=True,
unit='batch', file=sys.stdout)
file=sys.stdout)
setattr(self, f'{"test" if test else "val"}_progress_bar', pbar)

# run evaluation
Expand All @@ -319,9 +319,8 @@ def run_evaluation(self, test=False):
model.on_post_performance_check()

# add model specific metrics
tqdm_metrics = self.training_tqdm_dict
if not test:
self.main_progress_bar.set_postfix(**tqdm_metrics)
self.main_progress_bar.set_postfix(**self.training_tqdm_dict)

# close progress bar
if test:
Expand Down
22 changes: 4 additions & 18 deletions pytorch_lightning/trainer/trainer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -679,23 +679,9 @@ def training_tqdm_dict(self):
"""Read-only for tqdm metrics.
:return:
"""
tqdm_dict = {
'loss': '{0:.3f}'.format(self.avg_loss),
'batch_idx': '{}'.format(self.batch_idx),
}
ref_model = self.model if not self.data_parallel else self.model.module

if self.truncated_bptt_steps is not None:
tqdm_dict['split_idx'] = self.split_idx

if self.logger is not None and self.logger.version is not None:
tqdm_dict['v_num'] = self.logger.version

tqdm_dict.update(self.tqdm_metrics)

if self.on_gpu:
tqdm_dict['gpu'] = '{}'.format(torch.cuda.current_device())

return tqdm_dict
return dict(**ref_model.get_tqdm_dict(), **self.tqdm_metrics)
Copy link

@CarloLucibello CarloLucibello Jan 30, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would go with

if self.tqdm_metrics == None:
    return ref_model.get_tqdm_dict()
else:
    return self.tqdm_metrics

so that that the default dict can be completely overloaded

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But it is already can be completely overloaded :)
self.tqdm_metrics is not the default, it is totally determined by progress_bar dict from the training_end() and validation_end(). And model.get_tqdm_dict() adds some additional information like version number, gpu, etc. With this PR it is also totally controlled by the user.

So if you return an empty dict in model.get_tqdm_dict() and have no progress_bar dicts you will have a tqdm progress bar without any additional info.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right, sorry for the noise, I thought that get_tqdm_dict was not exposed. Is it discoverable? Should something be added somewhere in the docs?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, now it can be found as one of the methods of LightningModule. Maybe we can add some information about how progress bar is composed, but I don't understand where it can be inserted in the current docs.


@property
def tng_tqdm_dic(self):
Expand Down Expand Up @@ -853,7 +839,7 @@ def run_pretrain_routine(self, model):
pbar = tqdm(desc='Validation sanity check',
total=self.num_sanity_val_steps * len(self.get_val_dataloaders()),
leave=False, position=2 * self.process_position,
disable=not self.show_progress_bar, dynamic_ncols=True, unit='batch')
disable=not self.show_progress_bar, dynamic_ncols=True)
self.main_progress_bar = pbar
# dummy validation progress bar
self.val_progress_bar = tqdm(disable=True)
Expand All @@ -871,7 +857,7 @@ def run_pretrain_routine(self, model):

# init progress bar
pbar = tqdm(leave=True, position=2 * self.process_position,
disable=not self.show_progress_bar, dynamic_ncols=True, unit='batch',
disable=not self.show_progress_bar, dynamic_ncols=True,
file=sys.stdout)
self.main_progress_bar = pbar

Expand Down