Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

State saved with previous version of accelerate does not have such key in overrides #2924

Closed
breengles opened this issue Jul 9, 2024 · 1 comment

Comments

@breengles
Copy link

breengles commented Jul 9, 2024

Here is the particular line where it tries reaching step but it might not be there:

self.step = override_attributes["step"]

Seems like it has been added in 0.32.

Additionally, for some reason, I did not receive any logging messages even though the same exception should have been caught here as there is no step in states as well 🤷

    # Random states
    try:
        states = torch.load(input_dir.joinpath(f"{RNG_STATE_NAME}_{process_index}.pkl"))
        override_attributes["step"] = states["step"]
        random.setstate(states["random_state"])
        np.random.set_state(states["numpy_random_seed"])
        torch.set_rng_state(states["torch_manual_seed"])
        if is_xpu_available():
            torch.xpu.set_rng_state_all(states["torch_xpu_manual_seed"])
        else:
            torch.cuda.set_rng_state_all(states["torch_cuda_manual_seed"])
        if is_torch_xla_available():
            xm.set_rng_state(states["xm_seed"])
        logger.info("All random states loaded successfully")
    except Exception:
        logger.info("Could not load random states")
@breengles
Copy link
Author

Dup of #2923. Closing this issue

@breengles breengles changed the title States saved with previous version of accelerate does not have such key in overrides State saved with previous version of accelerate does not have such key in overrides Jul 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant