Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CPU-PSLIB] Add consistency insepection of op's embedding name and sparse table name in config_fleet.py, test=develop #34249

Merged
merged 3 commits into from
Jul 21, 2021

Conversation

WorgenZhang
Copy link
Contributor

@WorgenZhang WorgenZhang commented Jul 19, 2021

PR types

Function optimization

PR changes

Others

Describe

问题描述:

  1. 现在没有对fluid.layers.embedding的name与config_fleet.py中的sparse table name的一致性检查,导致用户遇到报错无法直接定位
  2. 之前的做法:当fluid.layers.embedding的name不在config_fleet.py中时,会对该embedding设置为默认的11维,如果用户设置的embedding的emb_size不是11维,会进行间接性报错(见下图),用户不清楚为什么

之前的报错日志:

  1. 报错1:sequence_conv_pool,fluid.layer.embedding传入的emb_size与默认配置的11维不符(由于name不在config_fleet.py里,所以默认配置)
    image
  2. 报错2: layers.fc, fc层传入的input是根据实际的emb_size获取的(11维),但fc层的参数shape是根据fluid.layer.embedding的shape获得的,因此mul时不一致而报错
    image

改动说明:
增加训练前的检查机制:1) 如果配置的fluid.layers.embedding的name不在config_fleet.py的config词典的key中,则进行warning提示,同时根据fluid.layers.embedding中的size配置生成一张新的sparse_table (原来是直接用默认值生成新的sparse_table,但这样会导致fluid.layers.embedding中的emb_size和实际的不符而报错) 2) 如果配置的fluid.layers.embedding的name在config_fleet.py的config词典的key中,但config_fleet.py中的sparse_embdex_dim + 3 != fluid.layers.embedding中的fea_dim,则直接进行报错

修改后报错日志:
image
image

…arse table name in config_fleet.py, test=develop
@paddle-bot-old
Copy link

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

Copy link
Contributor

@yaoxuefeng6 yaoxuefeng6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yaoxuefeng6 yaoxuefeng6 merged commit 2f76bb8 into PaddlePaddle:develop Jul 21, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants