fix RuntimeError: not allowed to set torch.backends.cudnn flags after disable_global_flags #3343
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
meta-cla
bot
added
the
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D81529616 |
TroyGarden
force-pushed
the
export-D81529616
branch
from
f7c2553 to
6abcf9b
Compare
TroyGarden
added a commit
to TroyGarden/torchrec
that referenced
this pull request
Sep 2, 2025
… disable_global_flags (meta-pytorch#3343) Summary: # context * after fix github CI workflow (GPU unit tests) we found lots of errors come from the same root cause: ``` torchrec/test_utils/__init__.py:129: in _wrapper return wrapped_func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ torchrec/distributed/test_utils/multi_process.py:131: in setUp torch.backends.cudnn.allow_tf32 = False ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <torch.backends.ContextProp object at 0x7f4e8bb3ba10> obj = <module 'torch.backends.cudnn' from '/opt/conda/envs/build_binary/lib/python3.11/site-packages/torch/backends/cudnn/__init__.py'> val = False def __set__(self, obj, val): if not flags_frozen(): self.setter(val) else: > raise RuntimeError( f"not allowed to set {obj.__name__} flags " "after disable_global_flags; please use flags() context manager instead" ) E RuntimeError: not allowed to set torch.backends.cudnn flags after disable_global_flags; please use flags() context manager instead ``` * it could be due to D78326114 according to D77758554 Rollback Plan: Differential Revision: D81529616
|
This pull request was exported from Phabricator. Differential Revision: D81529616 |
TroyGarden
added a commit
to TroyGarden/torchrec
that referenced
this pull request
Sep 2, 2025
… disable_global_flags (meta-pytorch#3343) Summary: # context * after fix github CI workflow (GPU unit tests) we found lots of errors come from the same root cause: ``` torchrec/test_utils/__init__.py:129: in _wrapper return wrapped_func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ torchrec/distributed/test_utils/multi_process.py:131: in setUp torch.backends.cudnn.allow_tf32 = False ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <torch.backends.ContextProp object at 0x7f4e8bb3ba10> obj = <module 'torch.backends.cudnn' from '/opt/conda/envs/build_binary/lib/python3.11/site-packages/torch/backends/cudnn/__init__.py'> val = False def __set__(self, obj, val): if not flags_frozen(): self.setter(val) else: > raise RuntimeError( f"not allowed to set {obj.__name__} flags " "after disable_global_flags; please use flags() context manager instead" ) E RuntimeError: not allowed to set torch.backends.cudnn flags after disable_global_flags; please use flags() context manager instead ``` * according to D77758554, the issue is due to D78326114 introducing `torch.testing._internal.common_utils` ``` # torch/testing/_internal/common_utils.py calls `disable_global_flags()` # workaround RuntimeError: not allowed to set ... after disable_global_flags ``` Differential Revision: D81529616
TroyGarden
force-pushed
the
export-D81529616
branch
from
6abcf9b to
193f1c4
Compare
|
This pull request was exported from Phabricator. Differential Revision: D81529616 |
TroyGarden
added a commit
to TroyGarden/torchrec
that referenced
this pull request
Sep 3, 2025
… disable_global_flags (meta-pytorch#3343) Summary: # context * after fix github CI workflow (GPU unit tests) we found lots of errors come from the same root cause: ``` torchrec/test_utils/__init__.py:129: in _wrapper return wrapped_func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ torchrec/distributed/test_utils/multi_process.py:131: in setUp torch.backends.cudnn.allow_tf32 = False ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <torch.backends.ContextProp object at 0x7f4e8bb3ba10> obj = <module 'torch.backends.cudnn' from '/opt/conda/envs/build_binary/lib/python3.11/site-packages/torch/backends/cudnn/__init__.py'> val = False def __set__(self, obj, val): if not flags_frozen(): self.setter(val) else: > raise RuntimeError( f"not allowed to set {obj.__name__} flags " "after disable_global_flags; please use flags() context manager instead" ) E RuntimeError: not allowed to set torch.backends.cudnn flags after disable_global_flags; please use flags() context manager instead ``` * according to D77758554, the issue is due to D78326114 introducing `torch.testing._internal.common_utils` ``` # torch/testing/_internal/common_utils.py calls `disable_global_flags()` # workaround RuntimeError: not allowed to set ... after disable_global_flags ``` Differential Revision: D81529616
TroyGarden
force-pushed
the
export-D81529616
branch
from
193f1c4 to
9396aa7
Compare
|
This pull request was exported from Phabricator. Differential Revision: D81529616 |
1 similar comment
|
This pull request was exported from Phabricator. Differential Revision: D81529616 |
TroyGarden
added a commit
to TroyGarden/torchrec
that referenced
this pull request
Sep 3, 2025
… disable_global_flags (meta-pytorch#3343) Summary: Pull Request resolved: meta-pytorch#3343 # context * after fix github CI workflow (GPU unit tests) we found lots of errors come from the same root cause: ``` torchrec/test_utils/__init__.py:129: in _wrapper return wrapped_func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ torchrec/distributed/test_utils/multi_process.py:131: in setUp torch.backends.cudnn.allow_tf32 = False ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <torch.backends.ContextProp object at 0x7f4e8bb3ba10> obj = <module 'torch.backends.cudnn' from '/opt/conda/envs/build_binary/lib/python3.11/site-packages/torch/backends/cudnn/__init__.py'> val = False def __set__(self, obj, val): if not flags_frozen(): self.setter(val) else: > raise RuntimeError( f"not allowed to set {obj.__name__} flags " "after disable_global_flags; please use flags() context manager instead" ) E RuntimeError: not allowed to set torch.backends.cudnn flags after disable_global_flags; please use flags() context manager instead ``` * according to D77758554, the issue is due to D78326114 introducing `torch.testing._internal.common_utils` ``` # torch/testing/_internal/common_utils.py calls `disable_global_flags()` # workaround RuntimeError: not allowed to set ... after disable_global_flags ``` Differential Revision: D81529616
TroyGarden
force-pushed
the
export-D81529616
branch
from
9396aa7 to
f6a1cb3
Compare
|
This pull request was exported from Phabricator. Differential Revision: D81529616 |
TroyGarden
added a commit
to TroyGarden/torchrec
that referenced
this pull request
Sep 3, 2025
… disable_global_flags (meta-pytorch#3343) Summary: Pull Request resolved: meta-pytorch#3343 # context * after fix github CI workflow (GPU unit tests) we found lots of errors come from the same root cause: ``` torchrec/test_utils/__init__.py:129: in _wrapper return wrapped_func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ torchrec/distributed/test_utils/multi_process.py:131: in setUp torch.backends.cudnn.allow_tf32 = False ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <torch.backends.ContextProp object at 0x7f4e8bb3ba10> obj = <module 'torch.backends.cudnn' from '/opt/conda/envs/build_binary/lib/python3.11/site-packages/torch/backends/cudnn/__init__.py'> val = False def __set__(self, obj, val): if not flags_frozen(): self.setter(val) else: > raise RuntimeError( f"not allowed to set {obj.__name__} flags " "after disable_global_flags; please use flags() context manager instead" ) E RuntimeError: not allowed to set torch.backends.cudnn flags after disable_global_flags; please use flags() context manager instead ``` * according to D77758554, the issue is due to D78326114 introducing `torch.testing._internal.common_utils` ``` # torch/testing/_internal/common_utils.py calls `disable_global_flags()` # workaround RuntimeError: not allowed to set ... after disable_global_flags ``` Differential Revision: D81529616
TroyGarden
force-pushed
the
export-D81529616
branch
from
f6a1cb3 to
c41589d
Compare
|
This pull request was exported from Phabricator. Differential Revision: D81529616 |
TroyGarden
added a commit
to TroyGarden/torchrec
that referenced
this pull request
Sep 3, 2025
… disable_global_flags (meta-pytorch#3343) Summary: Pull Request resolved: meta-pytorch#3343 # context * after fix github CI workflow (GPU unit tests) we found lots of errors come from the same root cause: ``` torchrec/test_utils/__init__.py:129: in _wrapper return wrapped_func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ torchrec/distributed/test_utils/multi_process.py:131: in setUp torch.backends.cudnn.allow_tf32 = False ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <torch.backends.ContextProp object at 0x7f4e8bb3ba10> obj = <module 'torch.backends.cudnn' from '/opt/conda/envs/build_binary/lib/python3.11/site-packages/torch/backends/cudnn/__init__.py'> val = False def __set__(self, obj, val): if not flags_frozen(): self.setter(val) else: > raise RuntimeError( f"not allowed to set {obj.__name__} flags " "after disable_global_flags; please use flags() context manager instead" ) E RuntimeError: not allowed to set torch.backends.cudnn flags after disable_global_flags; please use flags() context manager instead ``` * according to D77758554, the issue is due to D78326114 introducing `torch.testing._internal.common_utils` ``` # torch/testing/_internal/common_utils.py calls `disable_global_flags()` # workaround RuntimeError: not allowed to set ... after disable_global_flags ``` Differential Revision: D81529616
TroyGarden
force-pushed
the
export-D81529616
branch
from
c41589d to
b5783fb
Compare
… disable_global_flags (meta-pytorch#3343) Summary: Pull Request resolved: meta-pytorch#3343 # context * after fix github CI workflow (GPU unit tests) we found lots of errors come from the same root cause: ``` torchrec/test_utils/__init__.py:129: in _wrapper return wrapped_func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ torchrec/distributed/test_utils/multi_process.py:131: in setUp torch.backends.cudnn.allow_tf32 = False ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <torch.backends.ContextProp object at 0x7f4e8bb3ba10> obj = <module 'torch.backends.cudnn' from '/opt/conda/envs/build_binary/lib/python3.11/site-packages/torch/backends/cudnn/__init__.py'> val = False def __set__(self, obj, val): if not flags_frozen(): self.setter(val) else: > raise RuntimeError( f"not allowed to set {obj.__name__} flags " "after disable_global_flags; please use flags() context manager instead" ) E RuntimeError: not allowed to set torch.backends.cudnn flags after disable_global_flags; please use flags() context manager instead ``` * according to D77758554, the issue is due to D78326114 introducing `torch.testing._internal.common_utils` ``` # torch/testing/_internal/common_utils.py calls `disable_global_flags()` # workaround RuntimeError: not allowed to set ... after disable_global_flags ``` Reviewed By: aporialiao Differential Revision: D81529616
|
This pull request was exported from Phabricator. Differential Revision: D81529616 |
TroyGarden
force-pushed
the
export-D81529616
branch
from
b5783fb to
7ae565e
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
context
Rollback Plan:
Differential Revision: D81529616