Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not add special CUDNN search path rules for torch_python #37349

Closed
wants to merge 1 commit into from

Conversation

malfet
Copy link
Contributor

@malfet malfet commented Apr 27, 2020

Those rules never worked until #37275 and afterwards they are causing crashes in manywheels builds, because getting cudnn linked into libtorch_python and libtorch_cuda causes double-free exceptions, see: https://app.circleci.com/pipelines/github/pytorch/pytorch/160350/workflows/85696e1c-1e67-4780-8ceb-18bc0a614507/jobs/5254443

Test Plan: Enable manywheels build by temporarily enabling manywheels build on this PR and validate that it fixes the issue, see https://app.circleci.com/pipelines/github/pytorch/pytorch/160796/workflows/13227fbc-97c0-47f6-9a87-e840e1a4b5de/jobs/5267315/steps

@dr-ci
Copy link

dr-ci bot commented Apr 27, 2020

💊 Build failures summary and remediations

As of commit aa3dd9c (more details on the Dr. CI page):


  • 1/1 failures possibly* introduced in this PR
    • 1/1 non-CircleCI failure(s)

1 failure confirmed as flaky and can be ignored:

  • pytorch_windows_vs2019_py36_cuda10.1_build

ci.pytorch.org: 1 failed


This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker.

See how this bot performed.

This comment has been revised 7 times.

@malfet malfet requested review from kostmo, ezyang and seemethere April 27, 2020 18:50
@malfet malfet changed the title Malfet/test delete cuda rules Do not add special CUDNN search path rules for torch_python Apr 27, 2020
Copy link
Member

@kostmo kostmo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commit message typo Delte -> Delete

@malfet malfet force-pushed the malfet/test-delete-cuda-rules branch from a828558 to aa3dd9c Compare April 27, 2020 19:28
@malfet
Copy link
Contributor Author

malfet commented Apr 27, 2020

Updated PR by deleting 2nd commit.

@malfet malfet requested a review from kostmo April 27, 2020 19:28
@malfet
Copy link
Contributor Author

malfet commented Apr 27, 2020

Another interesting observation: before the change cudnn was actually linked statically into libtorch_python.so, which cause huge spike in binary size:
After this PR:

-rwxr-xr-x  1 root root  18M Apr 27 19:12 libtorch_python.so

Before:

-rwxr-xr-x  1 root root 236M Apr 27 19:32 libtorch_python.so

Copy link
Contributor

@facebook-github-bot facebook-github-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@malfet is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@malfet merged this pull request in 8dc5502.

@malfet malfet deleted the malfet/test-delete-cuda-rules branch April 27, 2020 22:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants