Skip to content

Commit

Permalink
Catch ConfigError when attempting to set num_parameters in WANDB
Browse files Browse the repository at this point in the history
The current version of the code catches AttributeError when
attempting to set the model's number of parameters in the
Weights & Biases config.

ConfigError is another exception that needs catching in the case
when the current number of model parameters differs from that
of the config. This happens, for example, on resuming from checkpoints if
extra layers have been added.

Example error message:

```
wandb.sdk.lib.config_util.ConfigError: Attempted to change value of key "model/num_parameters" from 0 to 700416
```

Since the current code already includes support for ignoring
AttributeError, it feels safe to add this extra catch.
  • Loading branch information
gheinrich committed Sep 12, 2024
1 parent d71d6cb commit d667888
Showing 1 changed file with 4 additions and 0 deletions.
4 changes: 4 additions & 0 deletions src/transformers/integrations/integration_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -853,6 +853,10 @@ def setup(self, args, state, model, **kwargs):
self._wandb.config["model/num_parameters"] = model.num_parameters()
except AttributeError:
logger.info("Could not log the number of model parameters in Weights & Biases.")
except self._wandb.sdk.lib.config_util.ConfigError:
logger.warning(
"A ConfigError was raised whilst setting the number of model parameters in Weights & Biases config."
)

# log the initial model architecture to an artifact
if self._log_model.is_enabled:
Expand Down

0 comments on commit d667888

Please sign in to comment.