You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Additionally, for some reason, I did not receive any logging messages even though the same exception should have been caught here as there is no step in states as well 🤷
# Random statestry:
states=torch.load(input_dir.joinpath(f"{RNG_STATE_NAME}_{process_index}.pkl"))
override_attributes["step"] =states["step"]
random.setstate(states["random_state"])
np.random.set_state(states["numpy_random_seed"])
torch.set_rng_state(states["torch_manual_seed"])
ifis_xpu_available():
torch.xpu.set_rng_state_all(states["torch_xpu_manual_seed"])
else:
torch.cuda.set_rng_state_all(states["torch_cuda_manual_seed"])
ifis_torch_xla_available():
xm.set_rng_state(states["xm_seed"])
logger.info("All random states loaded successfully")
exceptException:
logger.info("Could not load random states")
The text was updated successfully, but these errors were encountered:
breengles
changed the title
States saved with previous version of accelerate does not have such key in overrides
State saved with previous version of accelerate does not have such key in overrides
Jul 9, 2024
Here is the particular line where it tries reaching
step
but it might not be there:accelerate/src/accelerate/accelerator.py
Line 3147 in 5bdcd7e
Seems like it has been added in 0.32.
Additionally, for some reason, I did not receive any logging messages even though the same exception should have been caught here as there is no
step
instates
as well 🤷The text was updated successfully, but these errors were encountered: