Clarify batch size displayed when using DataParallel #24430
Conversation
|
The documentation is not available anymore as the PR was closed or merged. |
| logger.info(f" Instantaneous batch size per device = {self._train_batch_size:,}") | ||
| logger.info(f" Instantaneous batch size per device = {self.args.per_device_train_batch_size:,}") | ||
| if self.args.per_device_train_batch_size != self._train_batch_size: | ||
| logger.info(f" Training with DataParallel so batch size has been adjusted to: {self._train_batch_size:,}") | ||
| logger.info(f" Total train batch size (w. parallel, distributed & accumulation) = {total_train_batch_size:,}") |
There was a problem hiding this comment.
Thanks for care! #24345 seems resolved.
Sorry, I wrote below, but it seems not the case 😅 I didn't check gradient accumulation
So please ignore below
As I said #24345 (comment), I will not use DP anymore, but it seems to have bug around this logging part in case of DP.
e.g.) total_train_batch_size seems be larger than expected real value..?
May use
total_train_batch_size = self.args.per_device_train_batch_size * args.gradient_accumulation_steps * args.world_sizeat least in case of DP, instead of
transformers/src/transformers/trainer.py
Line 1665 in 8f093fb
..?
I'm not sure about this issue and my suggestion (as I'm not familiar yet with trainer core/internal) so by now I will not report any issue or make a PR, but you guys may care about this.
I would try to make new issue or PR if I have enough time for check 😄
What does this PR do?
As pointed out in #24345, the batch size displayed when using
DataParallelis unclear, this PR fixes that.Fixes #24345