Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove boost::variant #43100

Merged
merged 7 commits into from
Jun 28, 2022
Merged

Conversation

From00
Copy link
Contributor

@From00 From00 commented May 30, 2022

PR types

Others

PR changes

Others

Describe

为提升Paddle编译速度,尝试移除Paddle中老旧代码对boost::variant的使用,全量替换为paddle::variant,同时替换variant相关接口。

在全量替换过程中,暴露并在本PR中解决了以下问题:

  1. boost::apply_visitor不适用于paddle::variant,同步替换为paddle::visit,paddle::visit接口使用方式与std::visit对齐,不同于boost::apply_visitor
  2. paddle::variant的visit接口在默认#define MPARK_CPP14_CONSTEXPR的情况下,代码中实现的all函数需要接收std::initializer_list,但cpp14::lib::array在目前paddle的编译环境下无法转换为initializer_list,只有未#define MPARK_CPP14_CONSTEXPR的visit实现才可以顺利编译,因而本PR将无法编译的实现代码移除,只保留未#define MPARK_CPP14_CONSTEXPR情况下的visit接口实现70b3c4b67b5655291a40c56bde484348
  3. 在pybind/pybind_boost_headers.h中为boost::variant类型实现了type_caster模板特化,去除后可以通过编译,但运行时variant对象在C++端和python端的转换将出错,因而为paddle::variant同样实现了type_caster特化
  4. 许多地方调用get方法所用模板参数类型与variant声明的类型不一致,如box_decoder_and_assign_op.cu中尝试利用get<double>获取float类型的attr等,这在boost::variant中可以正确运行,但在paddle::variant中将触发编译检查拦截,本PR对这些不规范的写法一并进行了修改
  5. 全量替换后在Windows平台特定的运行环境下运行paddle\scripts\installation_validate.py,Parallel Executor执行完run函数返回类型为FetchResultType的fetch_data时程序会突然闪崩,退出码-1073741819。FetchResultType类型声明如下:
using FetchType = paddle::variant<LoDTensor, LoDTensorArray, framework::Vocab>;
using FetchList = std::vector<FetchType>;

using FetchUnmergedList = std::vector<std::vector<FetchType>>;
using FetchResultType = boost::variant<FetchList, FetchUnmergedList>;

该问题较棘手:
(1)按照开发者本地源码编译方式安装Paddle,无法复现此问题
(2)同样的测试机器按照Windows CI的环境部署和脚本运行,可复现错误
(3)出错位置代码语法没有明显问题,也捕捉不到代码层面的任何异常,怀疑是程序运行到函数返回处时被操作系统杀死,可能是底层存在一些bug导致此处函数返回后跳转地址错误,访问到非法地址而被OS拦截
(4)bug可能与requirement中的第三方依赖库有关(执行CI脚本后相同机器从稳定无法复现变成稳定复现,CI脚本运行后能遗留下来污染环境的只有pip第三方库的更新,但目前已难以回退到之前可以不出问题的环境);也可能是paddle的variant在嵌套使用时有问题(出问题的地方是嵌套variant,框架中别处都没有嵌套使用)
(5)对此问题目前没有太好的排查思路,因而对PE的FetchResultType先不做variant替换,后续重写PE相关函数接口,改成不用variant存储fetch_data,相关位置的get调用也临时性地使用裸的boost::get(此PR合入后提新PR修改PE并进行移除)

  1. 除了boost::variant,许多代码依赖#include "paddle/fluid/platform/variant.h"使用boost/variant.hpp中的其它名称,全量移除boost::variant后仍无法直接删除paddle/fluid/platform/variant.h,需等清理完其它boost中的符号使用后方可彻底清理对paddle/fluid/platform/variant.h的引用

本PR后续TODO:

  1. 重写PE的run接口,移除对FetchResultType类型的依赖
  2. 清除除variant外其它对boost库的使用
  3. 一些boost相关的符号命名替换,如BOOST_GET_CONST改成PADDLE_GET_CONST,boost相关注释清除等
  4. CMAKE第三方依赖中移除对boost库的加载

@paddle-bot-old
Copy link

Sorry to inform you that 0f14721's CIs have passed for more than 7 days. To prevent PR conflicts, you need to re-run all CIs manually.

Copy link
Contributor

@chenwhql chenwhql left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@JiabinYang JiabinYang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM for eager_generator

Copy link
Contributor

@zhiqiu zhiqiu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Collaborator

@raindrops2sea raindrops2sea left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@From00 From00 merged commit b3cf28f into PaddlePaddle:develop Jun 28, 2022
@From00 From00 mentioned this pull request Jul 11, 2022
@From00 From00 deleted the remove-boost-variant branch April 5, 2023 09:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants