Add warning to a known autograd issue on XLA backend. #35449

ailzhang · 2020-03-26T03:42:22Z

No description provided.

dr-ci · 2020-03-26T03:53:26Z

💊 CircleCI build failures summary and remediations

As of commit f9c1117 (more details on the Dr. CI page):

1/1 failures introduced in this PR

🕵️ 1 new failure recognized by patterns

The following build failures do not appear to be due to upstream breakages:

pytorch_linux_xenial_py3_6_gcc5_4_test (1/1)

Step: "Test" (full log | pattern match details) <confirmed not flaky by 2 failures>

Mar 26 20:59:54 [E request_callback_impl.cpp:94] Received error while processing request type 2: PickleError: ScriptModules cannot be deepcopied using copy.deepcopy or saved using torch.save. Mixed serialization of script and non-script modules is not supported. For purely script modules use my_script_module.save() instead.

Mar 26 20:59:54   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(86): serialize 
Mar 26 20:59:54   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(135): serialize 
Mar 26 20:59:54  
Mar 26 20:59:54 [E request_callback_impl.cpp:94] Received error while processing request type 2: PickleError: ScriptModules cannot be deepcopied using copy.deepcopy or saved using torch.save. Mixed serialization of script and non-script modules is not supported. For purely script modules use my_script_module.save(<filename>) instead. 
Mar 26 20:59:54  
Mar 26 20:59:54 At: 
Mar 26 20:59:54   /opt/conda/lib/python3.6/site-packages/torch/jit/__init__.py(1779): __getstate__ 
Mar 26 20:59:54   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(86): serialize 
Mar 26 20:59:54   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(135): serialize 
Mar 26 20:59:54  
Mar 26 20:59:54 [E request_callback_impl.cpp:94] Received error while processing request type 2: PickleError: ScriptModules cannot be deepcopied using copy.deepcopy or saved using torch.save. Mixed serialization of script and non-script modules is not supported. For purely script modules use my_script_module.save(<filename>) instead. 
Mar 26 20:59:54  
Mar 26 20:59:54 At: 
Mar 26 20:59:54   /opt/conda/lib/python3.6/site-packages/torch/jit/__init__.py(1779): __getstate__ 
Mar 26 20:59:54   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(86): serialize 
Mar 26 20:59:54   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(135): serialize 
Mar 26 20:59:54  
Mar 26 20:59:54 ok (1.118s) 
Mar 26 20:59:55   test_unexepected_kwarg_is_specified (__main__.JitRpcTestWithSpawn) ... ok (1.118s) 
Mar 26 20:59:56   test_user_rrefs_confirmed (__main__.JitRpcTestWithSpawn) ... ok (1.117s) 
Mar 26 20:59:57   test_user_rrefs_confirmed_remote (__main__.JitRpcTestWithSpawn) ... ok (1.117s)

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker.

See how this bot performed.

This comment has been revised 6 times.

albanD · 2020-03-26T14:16:59Z

torch/csrc/autograd/variable.cpp

@@ -384,6 +384,13 @@ unsigned VariableHooks::_register_hook(const Tensor& self, std::function<Tensor(
 }

 void handle_view_on_rebase(DifferentiableViewMeta* diff_view_meta, bool indirect) {
+  // TODO: Remove this warning once we allow XLA to workaround CopySlices.
+  if (diff_view_meta->base_.device().type() == c10::DeviceType::XLA) {
+    TORCH_WARN("Backward through inplace update on view tensors is WIP for XLA backwend. "


I would formulate this slightly differently maybe and adapt to the indirect flag:

When indirect=False, this is called because you are modifying inplace (right now) a view that requires gradients.

When indirect=True, this is called because another view of the same base was modified inplace (in the past) and this view requires gradients.

albanD

Looks much better, thanks for the changes!

albanD · 2020-03-26T18:28:02Z

torch/csrc/autograd/variable.cpp

+  if (diff_view_meta->base_.device().type() == c10::DeviceType::XLA) {
+    std::string msg;
+    if (indirect) {
+      msg = "This view requires gradients but its base or another view of the same base is modified inplace. ";


nit: Not sure here, isn't it "has been modified inplace"?

facebook-github-bot

@ailzhang has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot · 2020-03-27T12:22:15Z

@ailzhang merged this pull request in 2f6f178.

gchanan · 2020-03-27T14:49:11Z

torch/csrc/autograd/variable.cpp

+    } else {
+      msg = "This view requires gradients and it's being modified inplace. ";
+    }
+    msg = c10::str(msg, "Backward through inplace update on view tensors is WIP for XLA backwend. "


backwend -> backend.

Aha nice catch! Will send a fix!

gchanan · 2020-03-27T14:51:28Z

torch/csrc/autograd/variable.cpp

+      msg = "This view requires gradients and it's being modified inplace. ";
+    }
+    msg = c10::str(msg, "Backward through inplace update on view tensors is WIP for XLA backwend. "
+                   "Gradient might be wrong in certain cases. Running forward alone is fine. "


this prints in the forward? It's not clear what "alone is fine" means.

Is there anyway to 'tag' this and only print the warning in the backward?

Putting an example of this triggering would be helpful to see the effect.

Yes this prints in the forward when the inplace is done and only if inputs require gradients.
I think we don't want to delay the error as it would make it much harder for the user to find the faulty inplace.

We generally don't warn in the forward if it will cause a problem in the backward, right?

This seems like a good place for a mechanism you can turn on that warns/errors in the forward and the error in the backward tells you to turn it on.

Suggested rewording:

Running a backward pass through an inplace update on view tensors is a WIP for the XLA backend and may result in incorrect gradient computation in certain cases. Note this warning is being triggered on the inplace update (not the corresponding backward pass), and this update is safe if a backward pass is not run. To work around this limitation and to silence this warning, please replace the inplace operation by the corresponding out-of-place operation."

Thanks @gchanan ! It has been updated in #35543.

For these, we actually warn/error in the forward because we know that the backward will be wrong. Note that if you Tensor does not require gradients, then this won't be called. So people that do not use the autograd won't see it.

ailzhang requested review from albanD and apaszke as code owners March 26, 2020 03:42

albanD reviewed Mar 26, 2020

View reviewed changes

ailzhang force-pushed the xla_warning_copyslice branch from 94444e8 to be31227 Compare March 26, 2020 18:13

ailzhang requested a review from albanD March 26, 2020 18:18

albanD approved these changes Mar 26, 2020

View reviewed changes

Warn a known autograd issue on XLA backend.

f9c1117

ailzhang force-pushed the xla_warning_copyslice branch from be31227 to f9c1117 Compare March 26, 2020 18:30

facebook-github-bot reviewed Mar 26, 2020

View reviewed changes

facebook-github-bot closed this in 2f6f178 Mar 27, 2020

ailzhang mentioned this pull request Mar 27, 2020

Add warning to a known autograd issue on XLA backend. #35450

Merged

facebook-github-bot added the merged label Mar 27, 2020

gchanan reviewed Mar 27, 2020

View reviewed changes

mruberry added the Merged label Oct 28, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add warning to a known autograd issue on XLA backend. #35449

Add warning to a known autograd issue on XLA backend. #35449

ailzhang commented Mar 26, 2020

dr-ci bot commented Mar 26, 2020 •

edited

Loading

albanD Mar 26, 2020

albanD left a comment

albanD Mar 26, 2020

facebook-github-bot left a comment

facebook-github-bot commented Mar 27, 2020

gchanan Mar 27, 2020

ailzhang Mar 27, 2020

gchanan Mar 27, 2020

albanD Mar 27, 2020

gchanan Mar 27, 2020

gchanan Mar 27, 2020

ailzhang Mar 27, 2020

albanD Mar 27, 2020

Add warning to a known autograd issue on XLA backend. #35449

Add warning to a known autograd issue on XLA backend. #35449

Conversation

ailzhang commented Mar 26, 2020

dr-ci bot commented Mar 26, 2020 • edited Loading

💊 CircleCI build failures summary and remediations

🕵️ 1 new failure recognized by patterns

pytorch_linux_xenial_py3_6_gcc5_4_test (1/1)

Choose a reason for hiding this comment

albanD left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

facebook-github-bot left a comment

Choose a reason for hiding this comment

facebook-github-bot commented Mar 27, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dr-ci bot commented Mar 26, 2020 •

edited

Loading