Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Warn about pickle-ablity when using dp.map(some_local_function) ? #538

Closed
NicolasHug opened this issue Jun 23, 2022 · 1 comment
Closed

Warn about pickle-ablity when using dp.map(some_local_function) ? #538

NicolasHug opened this issue Jun 23, 2022 · 1 comment

Comments

@NicolasHug
Copy link
Member

torchdata issues a warning about pickle when we use lambdas (which is great!)
Another kind of function that isn't compatible with pickle are local functions. Would it be possible to throw the same warning there?

import torchdata
import pickle

def make_dp():

    def f(x):  # local function, not pickleable
        return x

    return torchdata.datapipes.iter.IterableWrapper(range(40)).map(f)

dp = make_dp()  # no warning

f = "/tmp/dp"
pickle.dump(dp, open(f, "wb"))  # fails
@ejguan
Copy link
Contributor

ejguan commented Jun 23, 2022

It makes sense to raise warning for local function. We might be able to use "<locals>" in fn.__qualname__ to determine it. Sending a patch.

@ejguan ejguan mentioned this issue Jun 23, 2022
Closed
pytorchmergebot pushed a commit to pytorch/pytorch that referenced this issue Jun 24, 2022
@ejguan @pytorchmergebot
Raise warning for unpickable local function (#80140)
4b75b7d 
Fixes pytorch/data#538

- Improve the validation function to raise warning about unpickable function when either lambda or local function is provided to `DataPipe`.
- The inner function from `functools.partial` object is extracted as well for validation
- Mimic the behavior of `pickle` module for local lambda function: It would only raise Error for the local function rather than `lambda` function. So, we will raise warning about local function not lambda function.
```py
>>> import pickle
>>> def fn():
...     lf = lambda x: x
...     pickle.dumps(lf)
>>> pickle.dumps(fn)
AttributeError: Can't pickle local object 'fn.<locals>.<lambda>'
```
Pull Request resolved: #80140
Approved by: https://github.com/VitalyFedyunin, https://github.com/NivekT
ejguan added a commit to ejguan/data that referenced this issue Jun 24, 2022
@ejguan @facebook-github-bot
Raise warning for unpickable local function
0eebef7 
Summary:
Fixes pytorch#538
- Improve the validation function to raise warning about unpickable function when either lambda or local function is provided to DataPipe.
- The inner function from functools.partial object is extracted as well for validation
- Mimic the behavior of pickle module for local lambda function: It would only raise Error for the local function rather than lambda function. So, we will raise warning about local function not lambda function.
```py

>>> import pickle
>>> def fn():
...     lf = lambda x: x
...     pickle.dumps(lf)
>>> pickle.dumps(fn)
AttributeError: Can't pickle local object 'fn.<locals>.<lambda>'
```

This Diff also fixes the Error introduced by pytorch/pytorch#79344

Differential Revision: D37417556

fbshipit-source-id: cf1df990b7ba9278d7eb9b62162edf3d573927ff
@ejguan ejguan mentioned this issue Jun 24, 2022
Closed
ejguan added a commit to ejguan/data that referenced this issue Jun 24, 2022
@ejguan @facebook-github-bot
Raise warning for unpickable local function (pytorch#547)
0d46374 
Summary:
Pull Request resolved: pytorch#547

Fixes pytorch#538
- Improve the validation function to raise warning about unpickable function when either lambda or local function is provided to DataPipe.
- The inner function from functools.partial object is extracted as well for validation
- Mimic the behavior of pickle module for local lambda function: It would only raise Error for the local function rather than lambda function. So, we will raise warning about local function not lambda function.
```py

>>> import pickle
>>> def fn():
...     lf = lambda x: x
...     pickle.dumps(lf)
>>> pickle.dumps(fn)
AttributeError: Can't pickle local object 'fn.<locals>.<lambda>'
```

This Diff also fixes the Error introduced by pytorch/pytorch#79344

Differential Revision: D37417556

fbshipit-source-id: 4877a7de9a2060f6da499dc26b7de64b4a200cd7
@ejguan ejguan mentioned this issue Jun 24, 2022
Closed
ejguan added a commit to ejguan/data that referenced this issue Jun 24, 2022
@ejguan @facebook-github-bot
Raise warning for unpickable local function (#80232)
0cc4106 
Summary:
X-link: pytorch/pytorch#80232

Pull Request resolved: pytorch#547

Fixes pytorch#538
- Improve the validation function to raise warning about unpickable function when either lambda or local function is provided to DataPipe.
- The inner function from functools.partial object is extracted as well for validation
- Mimic the behavior of pickle module for local lambda function: It would only raise Error for the local function rather than lambda function. So, we will raise warning about local function not lambda function.
```py

>>> import pickle
>>> def fn():
...     lf = lambda x: x
...     pickle.dumps(lf)
>>> pickle.dumps(fn)
AttributeError: Can't pickle local object 'fn.<locals>.<lambda>'
```

This Diff also fixes the Error introduced by pytorch/pytorch#79344

Differential Revision: D37417556

fbshipit-source-id: 7213ee84b34092e0c2cf293ff8bf1dc56659fc83
ejguan added a commit to ejguan/data that referenced this issue Jun 24, 2022
@ejguan @facebook-github-bot
Raise warning for unpickable local function (#80232)
16f963e 
Summary:
X-link: pytorch/pytorch#80232

Pull Request resolved: pytorch#547

Fixes pytorch#538
- Improve the validation function to raise warning about unpickable function when either lambda or local function is provided to DataPipe.
- The inner function from functools.partial object is extracted as well for validation
- Mimic the behavior of pickle module for local lambda function: It would only raise Error for the local function rather than lambda function. So, we will raise warning about local function not lambda function.
```py

>>> import pickle
>>> def fn():
...     lf = lambda x: x
...     pickle.dumps(lf)
>>> pickle.dumps(fn)
AttributeError: Can't pickle local object 'fn.<locals>.<lambda>'
```

This Diff also fixes the Error introduced by pytorch/pytorch#79344

Differential Revision: D37417556

fbshipit-source-id: c17e475bde703f2f55af140f2155d41efb45f049
ejguan added a commit to ejguan/pytorch that referenced this issue Jun 27, 2022
@ejguan @facebook-github-bot
Raise warning for unpickable local function (pytorch#80232)
373109c 
Summary:
Pull Request resolved: pytorch#80232

X-link: pytorch/data#547

Fixes pytorch/data#538
- Improve the validation function to raise warning about unpickable function when either lambda or local function is provided to DataPipe.
- The inner function from functools.partial object is extracted as well for validation
- Mimic the behavior of pickle module for local lambda function: It would only raise Error for the local function rather than lambda function. So, we will raise warning about local function not lambda function.
```py

>>> import pickle
>>> def fn():
...     lf = lambda x: x
...     pickle.dumps(lf)
>>> pickle.dumps(fn)
AttributeError: Can't pickle local object 'fn.<locals>.<lambda>'
```

This Diff also fixes the Error introduced by pytorch#79344

Test Plan:
```
buck test //caffe2/test:datapipe
buck test //pytorch/data/test:tests
```
Tested in OSS
```
# PT
pytest test/test_datapipe.py -v
# TD
pytest test/test_iterdatapipe.py -v
pytest test/test_mapdatapipe.py -v
pytest test/test_serialization.py -v
# TV
pytest test/test_prototype_builtin_datasets.py -v
```

Reviewed By: NivekT

Differential Revision: D37417556

fbshipit-source-id: 6fae4059285b8c742feda739cc5fe590b2e20c5e
ejguan added a commit to ejguan/data that referenced this issue Jun 27, 2022
@ejguan @facebook-github-bot
Raise warning for unpickable local function (#80232)
ad3474e 
Summary:
X-link: pytorch/pytorch#80232

Pull Request resolved: pytorch#547

Fixes pytorch#538
- Improve the validation function to raise warning about unpickable function when either lambda or local function is provided to DataPipe.
- The inner function from functools.partial object is extracted as well for validation
- Mimic the behavior of pickle module for local lambda function: It would only raise Error for the local function rather than lambda function. So, we will raise warning about local function not lambda function.
```py

>>> import pickle
>>> def fn():
...     lf = lambda x: x
...     pickle.dumps(lf)
>>> pickle.dumps(fn)
AttributeError: Can't pickle local object 'fn.<locals>.<lambda>'
```

This Diff also fixes the Error introduced by pytorch/pytorch#79344

Reviewed By: NivekT

Differential Revision: D37417556

fbshipit-source-id: 6babb73a7daad470ec93c3bd0a0c08d71849d3c8
pytorchmergebot pushed a commit to pytorch/pytorch that referenced this issue Jun 27, 2022
@ejguan @pytorchmergebot
Raise warning for unpickable local function (#547) (#80232)
3d218e1 
Summary:
X-link: pytorch/data#547

Fixes pytorch/data#538
- Improve the validation function to raise warning about unpickable function when either lambda or local function is provided to DataPipe.
- The inner function from functools.partial object is extracted as well for validation
- Mimic the behavior of pickle module for local lambda function: It would only raise Error for the local function rather than lambda function. So, we will raise warning about local function not lambda function.
```py

>>> import pickle
>>> def fn():
...     lf = lambda x: x
...     pickle.dumps(lf)
>>> pickle.dumps(fn)
AttributeError: Can't pickle local object 'fn.<locals>.<lambda>'
```

This Diff also fixes the Error introduced by #79344

Test Plan:
CI on PyTorch and TorchData
Manually validated the tests from TorchVision

Differential Revision: D37417556

Pull Request resolved: #80232
Approved by: https://github.com/NivekT
facebook-github-bot pushed a commit to pytorch/pytorch that referenced this issue Jun 27, 2022
@ejguan @facebook-github-bot
Raise warning for unpickable local function (#80232)
3d96af7 
Summary:
Pull Request resolved: #80232

X-link: pytorch/data#547

Fixes pytorch/data#538
- Improve the validation function to raise warning about unpickable function when either lambda or local function is provided to DataPipe.
- The inner function from functools.partial object is extracted as well for validation
- Mimic the behavior of pickle module for local lambda function: It would only raise Error for the local function rather than lambda function. So, we will raise warning about local function not lambda function.
```py

>>> import pickle
>>> def fn():
...     lf = lambda x: x
...     pickle.dumps(lf)
>>> pickle.dumps(fn)
AttributeError: Can't pickle local object 'fn.<locals>.<lambda>'
```

This Diff also fixes the Error introduced by #79344

Test Plan:
```
buck test //caffe2/test:datapipe
buck test //pytorch/data/test:tests
```
Tested in OSS
```
# PT
pytest test/test_datapipe.py -v
# TD
pytest test/test_iterdatapipe.py -v
pytest test/test_mapdatapipe.py -v
pytest test/test_serialization.py -v
# TV
pytest test/test_prototype_builtin_datasets.py -v
```

Reviewed By: NivekT

Differential Revision: D37417556

fbshipit-source-id: 388d04004da43aaf14729cbe465a9247bef6c780
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@NicolasHug @ejguan