How to remove the randomness to get reproducible runs

[a-mos]

Hello! I am trying to get fully-reproducible pipeline, however with all seeds fixed i got different grad values on first iteration in single_circle.py app even with same loss value:

Fixing all seeds:

import random
import os

seed = 0
torch.set_printoptions(precision=16)
os.environ['PYTHONHASHSEED'] = str(seed)
torch.manual_seed(seed)
random.seed(seed)
np.random.seed(seed)
torch.backends.cudnn.benchmark = False
torch.use_deterministic_algorithms(True)
os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.enabled = False
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
torch.set_deterministic_debug_mode(2)

Run1 python single_circle.py

iteration: 0
loss: 6360.21044921875
radius.grad: tensor(-6679.3833007812500000)
center.grad: tensor([-16965.3789062500000000,   8691.6396484375000000])

Run2 python single_circle.py

iteration: 0
loss: 6360.21044921875
radius.grad: tensor(-6679.9208984375000000)
center.grad: tensor([-16965.9726562500000000,   8691.4550781250000000])

[tanguymagne]

Hello,
I am facing the same issue. Did you manage to solve it?
Thank you very much