Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

paddle.save/paddle.static.save 升级pickle的版本 #31044

Merged
merged 4 commits into from
Feb 23, 2021
Merged

paddle.save/paddle.static.save 升级pickle的版本 #31044

merged 4 commits into from
Feb 23, 2021

Conversation

hbwx24
Copy link
Contributor

@hbwx24 hbwx24 commented Feb 19, 2021 •
edited
Loading

PR types

Others

PR changes

APIs

Describe

一、主要修改

  • paddle.save/static.save指定pickle版本参数protocol(2<= pickle_protocol <=4)，默认pickle_protocol=2。

二、Example：

layer = Layer()
save_dict = layer.state_dict()
path = os.path.join("test_paddle_save_load_pickle_protocol", "layer")
paddle.save(save_dict, path, pickle_protocol=3)
load_state_dict=paddle.load(path)

三、TODO：

  • 保存/加载Var相关的复杂对象。
  • 保存Layer等。

四、注意：

  • pickle协议版本 2 是在 Python 2.3 中添加的。
  • pickle协议版本 3 是在 Python 3.0 中添加的。
  • pickle协议版本 4 是在 Python 3.4 中添加的。
  • pickle协议版本 5 是在 Python 3.8 中添加的。

五、文档修改：
PaddlePaddle/docs#3258
PaddlePaddle/docs#3272
六、文档预览：
paddle.save
image
image

paddle.static.save
image
image

@paddle-bot-old
Copy link

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

chenwhql
chenwhql reviewed Feb 19, 2021
View reviewed changes
Copy link
Contributor

@chenwhql chenwhql left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

还需要更新文档

python/paddle/fluid/io.py Outdated
@@ -1799,6 +1806,14 @@ def save(program, model_path):
assert base_name != "", \
"The input model_path MUST be format of dirname/filename [dirname\\filename in Windows system], but received model_path is empty string."

if not isinstance(protocol, int):
raise ValueError("The 'protocol' should be `int`,but received {}".
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

,but中间加空格

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx,done.

python/paddle/fluid/io.py Outdated
format(type(protocol)))

if protocol < 2 or protocol > 4:
raise ValueError("Expected 1<'protocol'<5 ,but received protocol={}".
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

,but -> , but

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx,done.

python/paddle/framework/io.py Outdated
@@ -254,26 +255,36 @@ def save(obj, path):
"[dirname\\filename in Windows system], but received "
"filename is empty string.")

if not isinstance(protocol, int):
raise ValueError("The 'protocol' MUST be `int`,but received {}".format(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same above

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx,done.

python/paddle/framework/io.py Outdated
type(protocol)))

if protocol < 2 or protocol > 4:
raise ValueError("Expected 1<'protocol'<5 ,but received protocol={}".
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same above

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx,done.

@hbwx24 hbwx24 mentioned this pull request Feb 19, 2021
Merged
@hbwx24 hbwx24 closed this Feb 19, 2021
@hbwx24 hbwx24 reopened this Feb 19, 2021
@hbwx24 hbwx24 closed this Feb 19, 2021
@hbwx24 hbwx24 reopened this Feb 19, 2021
@hbwx24 hbwx24 closed this Feb 19, 2021
@hbwx24 hbwx24 reopened this Feb 19, 2021
@hbwx24 hbwx24 closed this Feb 19, 2021
@hbwx24 hbwx24 reopened this Feb 19, 2021
@hbwx24 hbwx24 closed this Feb 19, 2021
@hbwx24 hbwx24 reopened this Feb 19, 2021
@hbwx24 hbwx24 force-pushed the save_load/save_pickle_protocol branch from 810763d to 5bdf1db Compare February 19, 2021 11:38
@hbwx24 hbwx24 closed this Feb 20, 2021
@hbwx24 hbwx24 reopened this Feb 20, 2021
chenwhql
chenwhql reviewed Feb 20, 2021
View reviewed changes
python/paddle/fluid/io.py Outdated
@@ -1771,6 +1778,8 @@ def save(program, model_path):
Args:
program(Program) : The program to saved.
model_path(str): the file prefix to save the program. The format is "dirname/file_prefix". If file_prefix is empty str. A exception will be raised
pickle_protocol(int, optional): The protocol version of pickle module must be greater than 1 and less than 5.
Default: None
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Default是2？

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx，done

python/paddle/framework/io.py Outdated
@@ -218,6 +219,8 @@ def save(obj, path):
obj(Object) : The object to be saved.
path(str) : The path of the object to be saved.
If saved in the current directory, the input path string will be used as the file name.
pickle_protocol(int, optional): The protocol version of pickle module must be greater than 1 and less than 5.
Default: None
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

同上

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done，thx。

chenwhql
chenwhql previously approved these changes Feb 20, 2021
View reviewed changes
Copy link
Contributor

@chenwhql chenwhql left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

TCChenlong
TCChenlong previously approved these changes Feb 22, 2021
View reviewed changes
Copy link
Contributor

@TCChenlong TCChenlong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

lanxianghit
lanxianghit reviewed Feb 22, 2021
View reviewed changes
python/paddle/fluid/io.py
i * MAX_NUMBER_OF_ELEMENT:MAX_NUMBER_OF_ELEMENT * (i + 1
)]

if 1 < protocol < 4:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

对特定版本的特殊处理逻辑最好注释解释一下，方便其他人理解以及后续维护，可以在后续pr再补充

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

lanxianghit
lanxianghit previously approved these changes Feb 22, 2021
View reviewed changes
TCChenlong
TCChenlong previously approved these changes Feb 23, 2021
View reviewed changes
Copy link
Contributor

@TCChenlong TCChenlong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hbwx24 @lanxianghit
Update python/paddle/fluid/io.py
3ed1253 
Co-authored-by: lanxianghit <47554610+lanxianghit@users.noreply.github.com>
lanxianghit
lanxianghit approved these changes Feb 23, 2021
View reviewed changes
Copy link
Contributor

@lanxianghit lanxianghit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

TCChenlong
TCChenlong approved these changes Feb 23, 2021
View reviewed changes
Copy link
Contributor

@TCChenlong TCChenlong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hbwx24 hbwx24 closed this Feb 23, 2021
@hbwx24 hbwx24 reopened this Feb 23, 2021
@chenwhql chenwhql merged commit ee1801c into PaddlePaddle:develop Feb 23, 2021
hbwx24 added a commit to hbwx24/Paddle that referenced this pull request Feb 23, 2021
@hbwx24 @lanxianghit
Save load/save pickle protocol (PaddlePaddle#31044)
6d7ca4c 
* add default argument  for paddle.save/static.save

* edit documentation of

* Add comments for special processing for protocol=2 and protocol=3.

* Update python/paddle/fluid/io.py

Co-authored-by: lanxianghit <47554610+lanxianghit@users.noreply.github.com>

Co-authored-by: lanxianghit <47554610+lanxianghit@users.noreply.github.com>
@hbwx24 hbwx24 mentioned this pull request Feb 23, 2021
Merged
@hbwx24 hbwx24 changed the title Save load/save pickle protocol paddle.save/paddle.static.save 升级pickle的版本 Feb 23, 2021
lanxianghit pushed a commit that referenced this pull request Feb 23, 2021
@hbwx24
[cherry-pick] paddle.save/paddle.static.save 升级pickle的版本。 (#31044) (#…
5d045b9 
…31140)

paddle.save/static.save指定pickle版本参数protocol(2<= pickle_protocol <=4)，默认pickle_protocol=2。
原始PR：#31044
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@TCChenlong TCChenlong TCChenlong approved these changes

@lanxianghit lanxianghit lanxianghit approved these changes

@chenwhql chenwhql chenwhql left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@hbwx24 @TCChenlong @chenwhql @lanxianghit