Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix buffer overflow for unzip with columns_to_skip #658

Closed
wants to merge 6 commits into from
Closed

Fix buffer overflow for unzip with columns_to_skip #658

wants to merge 6 commits into from

Conversation

ejguan
Copy link
Contributor

@ejguan ejguan commented Jul 14, 2022

When unzip has columns_to_skip, we should only create ChildDataPipe corresponding to instance_ids. And Unzipper should also handle mapping instance_ids to 0-based indices.

This is one of the root causes for the issue reported internally. See: https://fburl.com/2k0et1gv

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 14, 2022
@ejguan ejguan requested a review from NivekT July 14, 2022 20:53
@facebook-github-bot
Copy link
Contributor

@ejguan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@ejguan
Copy link
Contributor Author

ejguan commented Jul 14, 2022

I will add tests tmrw

NivekT
NivekT approved these changes Jul 15, 2022
View reviewed changes
Copy link
Contributor

@NivekT NivekT left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Just need tests. Can you test that the buffer properly clears when columns_to_skip is used with unzip? Thanks!

torchdata/datapipes/iter/util/unzipper.py Show resolved Hide resolved
@facebook-github-bot
Copy link
Contributor

@ejguan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@ejguan
Copy link
Contributor Author

ejguan commented Jul 18, 2022 •
edited
Loading

This PR requires pytorch/pytorch#81502 landed and nightly released.

@facebook-github-bot
Copy link
Contributor

@ejguan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

NivekT
NivekT approved these changes Jul 19, 2022
View reviewed changes
torchdata/datapipes/iter/util/unzipper.py Outdated Show resolved Hide resolved
torchdata/datapipes/iter/util/unzipper.py Outdated Show resolved Hide resolved
@facebook-github-bot
Copy link
Contributor

@ejguan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

1 similar comment
@facebook-github-bot
Copy link
Contributor

@ejguan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@NivekT NivekT NivekT approved these changes

@VitalyFedyunin VitalyFedyunin Awaiting requested review from VitalyFedyunin

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ejguan @facebook-github-bot @NivekT