昇腾和寒武纪相关代码退场 Tracking Issue #52319

GreatV · 2023-03-29T13:39:01Z

一、项目说明
用于跟踪昇腾和寒武纪相关代码退场并记录该计划下各个子任务的计划和进度。

二、主要工作 (18/18)

按 merge 的时间顺序，排名不分先后： @KimBioInfoStudio (5) @mrcangye (2) @jjyaoao (3) @Young-Flash (1) @Liyulingyue (3) @engineer1109 (1) @GreatV (2)

任务编号	任务内容	认领人	PR链接
1	删除 `paddle/fluid/platform/device/mlu` 目录✅(2023/3/31)	@mrcangye	#52382
2	删除 `paddle/fluid/operators/_mlu.` 文件✅(2023/4/3)	@Young-Flash	#52435
3	清理 `WITH_MLU` 相关代码✅(2023/4/13)	@GreatV @jjyaoao	#52546✅ #52825✅
4	清理 `is_compiled_with_mlu` 相关代码✅(2023/4/3)	@KimBioInfoStudio	#52378
5	删除 `paddle/fluid/operators/_npu.`、清理 `WITH_ASCEND_INT64` 相关代码✅(2023/4/11)	@jjyaoao @mrcangye	#52673 #52674 #52675 #52676 #52677 #52678 #52679 #53342
6	删除 `/paddle/fluid/platform/device/npu` 目录✅(2023/4/1)	@jjyaoao	#52384
7	清理 `WITH_ASCEND_CL PADDLE_WITH_ASCEND_CL WITH_ASCEND_CXX11` 相关代码✅(2023/4/13)	@engineer1109 @Liyulingyue @jjyaoao	#52448✅ #52535✅ #52612✅ #52825✅
8	清理 `is_compiled_with_npu` 相关代码✅(2023/4/6)	@KimBioInfoStudio	#52385
9	清理 `PADDLE_WITH_ASCEND` 相关代码✅(2023/4/13)	@mrcangye @jjyaoao	#52403✅ #52830✅
10	清理 `WITH_ASCEND` 相关代码✅(2023/4/10)	@Liyulingyue	#52669
11	清理`paddle_build.sh` 脚本中所有的`WITH_MLU` `WITH_ASCEND*`关键字✅(2023/3/31)	@KimBioInfoStudio	#52373
12	清理 `CNCL` 相关代码✅(2023/4/17)	@KimBioInfoStudio	#52811✅ #52434✅ #52939
13	清理 `HCCL` 相关代码✅(2023/4/20)	@Liyulingyue	#52798✅ #52934✅ #52942✅ #53058✅
14	清理 Dockerfile✅(2023/4/4)	@KimBioInfoStudio	#52474
15	清理 `is_mlu_place` `is_npu_place` `is_npu_pinned_place` `SupportNPU` `SupportMLU` 相关代码✅(2023/4/28)	@jjyaoao @Liyulingyue	#52911✅ #53105✅ #53391✅
16	清理 ASCEND 相关代码 ✅(2023/4/24)	@GreatV	#53046✅ #53132✅
17	清理 MLU 相关代码✅(2023/4/24)	@Liyulingyue	#53007✅ #53194✅
18	清理 NPU 相关代码✅(2023/5/18)	@KimBioInfoStudio @Liyulingyue @jjyaoao	#53469✅ #53566✅ #53568✅ #53677✅ #53699✅ #53869✅ #53870✅

需要清理的具体文件可参考此链接。

The text was updated successfully, but these errors were encountered:

GreatV · 2023-03-30T07:56:43Z

1-3. 清理`WITH_MLU`相关代码，删除 `paddle/fluid/platform/device/mlu`目录，删除 `paddle/fluid/operators/*_mlu.*` 文件。具体需清理的文件如下所示：

- `./CMakeLists.txt`
- `./test/CMakeLists.txt`
- `./cmake/operators.cmake`
- `./cmake/third_party.cmake`
- `./cmake/configure.cmake`
- `./cmake/neuware.cmake`
- `./python/CMakeLists.txt`
- `./python/paddle/device/__init__.py`
- `./paddle/fluid/imperative/CMakeLists.txt`
- `./paddle/fluid/imperative/prepared_operator.cc`
- `./paddle/fluid/imperative/tracer.cc`
- `./paddle/fluid/imperative/gradient_accumulator.cc`
- `./paddle/fluid/imperative/amp_auto_cast.cc`
- `./paddle/fluid/memory/memcpy.h`
- `./paddle/fluid/memory/memcpy.cc`
- `./paddle/fluid/memory/allocation/buddy_allocator.cc`
- `./paddle/fluid/memory/allocation/buddy_allocator_test.cc`
- `./paddle/fluid/memory/allocation/allocator_facade.cc`
- `./paddle/fluid/memory/allocation/system_allocator.cc`
- `./paddle/fluid/memory/allocation/system_allocator_test.cc`
- `./paddle/fluid/memory/allocation/naive_best_fit_allocator.cc`
- `./paddle/fluid/memory/allocation/system_allocator.h`
- `./paddle/fluid/memory/allocation/naive_best_fit_allocator_test.cc`
- `./paddle/fluid/operators/reader/buffered_reader.cc`
- `./paddle/fluid/operators/reader/buffered_reader.h`
- `./paddle/fluid/operators/where_op_mlu.cc`
- `./paddle/fluid/operators/utils.h`
- `./paddle/fluid/operators/CMakeLists.txt`
- `./paddle/fluid/operators/cast_op.cc`
- `./paddle/fluid/operators/reduce_ops/reduce_op_mlu.h`
- `./paddle/fluid/operators/mlu/CMakeLists.txt`
- `./paddle/fluid/operators/squeeze_op_mlu.cc`
- `./paddle/fluid/operators/math/CMakeLists.txt`
- `./paddle/fluid/operators/math/concat_and_split.cc`
- `./paddle/fluid/operators/expand_v2_op.h`
- `./paddle/fluid/operators/unsqueeze_op_mlu.cc`
- `./paddle/fluid/operators/elementwise/elementwise_max_op_mlu.cc`
- `./paddle/fluid/operators/elementwise/elementwise_mlu.h`
- `./paddle/fluid/operators/tile_op_mlu.cc`
- `./paddle/fluid/operators/coalesce_tensor_op.cc`
- `./paddle/fluid/operators/shape_op_mlu.cc`
- `./paddle/fluid/operators/expand_v2_op_mlu.cc`
- `./paddle/fluid/operators/detection/CMakeLists.txt`
- `./paddle/fluid/operators/softmax_with_cross_entropy_op.cc`
- `./paddle/fluid/framework/garbage_collector.cc`
- `./paddle/fluid/framework/executor.cc`
- `./paddle/fluid/framework/parallel_executor.cc`
- `./paddle/fluid/framework/tensor_util.h`
- `./paddle/fluid/framework/phi_utils.cc`
- `./paddle/fluid/framework/tensor_util.cc`
- `./paddle/fluid/framework/operator.cc`
- `./paddle/fluid/framework/garbage_collector.h`
- `./paddle/fluid/platform/init.cc`
- `./paddle/fluid/platform/CMakeLists.txt`
- `./paddle/fluid/platform/device_context.cc`
- `./paddle/fluid/platform/device_context.h`
- `./paddle/fluid/platform/profiler_helper.h`
- `./paddle/fluid/platform/init_test.cc`
- `./paddle/fluid/platform/device/CMakeLists.txt`
- `./paddle/fluid/platform/device/mlu/mlu_info.h`
- `./paddle/fluid/platform/device/mlu/CMakeLists.txt`
- `./paddle/fluid/platform/device/mlu/device_context.cc`
- `./paddle/fluid/platform/device/mlu/device_context.h`
- `./paddle/fluid/platform/device/mlu/mlu_resource_pool.cc`
- `./paddle/fluid/platform/device/mlu/mlu_stream.h`
- `./paddle/fluid/platform/device/mlu/enforce_test.cc`
- `./paddle/fluid/platform/device/mlu/mlu_resource_pool.h`
- `./paddle/fluid/platform/device/mlu/enforce.h`
- `./paddle/fluid/platform/device/mlu/mlu_stream.cc`
- `./paddle/fluid/platform/device/device_wrapper.h`
- `./paddle/fluid/platform/stream_callback_manager.cc`
- `./paddle/fluid/platform/profiler/profiler.cc`
- `./paddle/fluid/platform/profiler/mlu/CMakeLists.txt`
- `./paddle/fluid/platform/profiler/mlu/cnpapi_data_process.h`
- `./paddle/fluid/platform/profiler/mlu/mlu_tracer.h`
- `./paddle/fluid/platform/profiler/mlu/mlu_tracer.cc`
- `./paddle/fluid/platform/profiler/mlu/cnpapi_data_process.cc`
- `./paddle/fluid/platform/profiler/chrometracing_logger.cc`
- `./paddle/fluid/platform/place.h`
- `./paddle/fluid/pybind/tensor_py.h`
- `./paddle/fluid/pybind/tensor.cc`
- `./paddle/fluid/pybind/parallel_executor.cc`
- `./paddle/fluid/pybind/pybind.cc`
- `./paddle/fluid/pybind/place.cc`
- `./paddle/phi/kernels/funcs/strided_memcpy.h`
- `./paddle/phi/kernels/funcs/activation_functor.h`
- `./paddle/phi/core/utils/visit_place.h`
- `./paddle/phi/core/flags.cc`
- `./paddle/phi/backends/device_memory_aligment.h`
- `./paddle/phi/backends/mlu/mlu_info.h`
- `./paddle/phi/backends/context_pool.cc`

4. 清理 `is_compiled_with_mlu` 相关代码，具体如下：

- `./tools/get_quick_disable_lt.py`
- `./python/paddle/distributed/collective.py`
- `./python/paddle/distributed/launch/context/device.py`
- `./python/paddle/distributed/fleet/launch.py`
- `./python/paddle/distributed/fleet/launch_utils.py`
- `./python/paddle/distributed/parallel.py`
- `./python/paddle/__init__.py`
- `./python/paddle/fluid/dygraph/varbase_patch_methods.py`
- `./python/paddle/fluid/tests/unittests/test_dist_base.py`
- `./python/paddle/fluid/tests/unittests/eager_op_test.py`
- `./python/paddle/fluid/framework.py`
- `./python/paddle/fluid/executor.py`
- `./python/paddle/static/amp/fp16_lists.py`
- `./python/paddle/device/__init__.py`
- `./paddle/fluid/pybind/pybind.cc`

5-7. 清理`WITH_ASCEND_CL PADDLE_WITH_ASCEND_CL WITH_ASCEND_CXX11 WITH_ASCEND_INT64`相关代码，删除 `paddle/fluid/operators/*_npu.*`，删除 `/paddle/fluid/platform/device/npu`目录。

-  `./tools/coverage/paddle_coverage_new.sh`
-  `./tools/coverage/paddle_coverage.sh`
-  `./CMakeLists.txt`
-  `./test/asp/CMakeLists.txt`
-  `./cmake/operators.cmake`
-  `./cmake/inference_lib.cmake`
-  `./cmake/third_party.cmake`
-  `./cmake/configure.cmake`
-  `./cmake/external/warpctc.cmake`
-  `./cmake/external/protobuf.cmake`
-  `./cmake/external/threadpool.cmake`
-  `./cmake/external/gloo.cmake`
-  `./cmake/external/ascend.cmake`
-  `./cmake/flags.cmake`
-  `./python/CMakeLists.txt`
-  `./python/paddle/fluid/tests/unittests/CMakeLists.txt`
-  `./python/paddle/fluid/tests/unittests/collective/CMakeLists.txt`
-  `./python/paddle/fluid/tests/unittests/collective/fleet/CMakeLists.txt`
-  `./python/paddle/fluid/tests/unittests/distributed_passes/CMakeLists.txt`
-  `./python/paddle/device/__init__.py`
-  `./paddle/fluid/imperative/CMakeLists.txt`
-  `./paddle/fluid/imperative/prepared_operator.cc`
-  `./paddle/fluid/imperative/tracer.cc`
-  `./paddle/fluid/imperative/heter_ccl_context.cc`
-  `./paddle/fluid/imperative/gradient_accumulator.cc`
-  `./paddle/fluid/imperative/amp_auto_cast.cc`
-  `./paddle/fluid/imperative/heter_ccl_context.h`
-  `./paddle/fluid/imperative/hccl_context.h`
-  `./paddle/fluid/imperative/reducer.h`
-  `./paddle/fluid/imperative/reducer.cc`
-  `./paddle/fluid/memory/memcpy.cc`
-  `./paddle/fluid/memory/allocation/buddy_allocator.cc`
-  `./paddle/fluid/memory/allocation/buddy_allocator_test.cc`
-  `./paddle/fluid/memory/allocation/allocator_facade.h`
-  `./paddle/fluid/memory/allocation/npu_pinned_allocator.h`
-  `./paddle/fluid/memory/allocation/CMakeLists.txt`
-  `./paddle/fluid/memory/allocation/allocator_facade.cc`
-  `./paddle/fluid/memory/allocation/system_allocator.cc`
-  `./paddle/fluid/memory/allocation/system_allocator_test.cc`
-  `./paddle/fluid/memory/allocation/npu_pinned_allocator.cc`
-  `./paddle/fluid/memory/allocation/naive_best_fit_allocator.cc`
-  `./paddle/fluid/memory/allocation/system_allocator.h`
-  `./paddle/fluid/memory/allocation/naive_best_fit_allocator_test.cc`
-  `./paddle/fluid/operators/reader/buffered_reader.cc`
-  `./paddle/fluid/operators/reader/buffered_reader.h`
-  `./paddle/fluid/operators/memcpy_op.cc`
-  `./paddle/fluid/operators/memcpy_h2d_op.cc`
-  `./paddle/fluid/operators/utils.h`
-  `./paddle/fluid/operators/scatter_op_npu.cc`
-  `./paddle/fluid/operators/memcpy_d2h_op.cc`
-  `./paddle/fluid/operators/CMakeLists.txt`
-  `./paddle/fluid/operators/copy_cross_scope_test.cc`
-  `./paddle/fluid/operators/expand_op.h`
-  `./paddle/fluid/operators/reduce_ops/CMakeLists.txt`
-  `./paddle/fluid/operators/collective/c_sync_calc_stream_op.h`
-  `./paddle/fluid/operators/collective/partial_recv_op.cc`
-  `./paddle/fluid/operators/collective/partial_send_op.cc`
-  `./paddle/fluid/operators/collective/c_allreduce_max_op_npu_test.cc`
-  `./paddle/fluid/operators/collective/c_reduce_sum_op_npu_test.cc`
-  `./paddle/fluid/operators/collective/CMakeLists.txt`
-  `./paddle/fluid/operators/collective/checknumeric_npu_test.cc`
-  `./paddle/fluid/operators/collective/c_allgather_op_npu_test.cc`
-  `./paddle/fluid/operators/collective/c_reduce_op.h`
-  `./paddle/fluid/operators/collective/gen_hccl_id_op.cc`
-  `./paddle/fluid/operators/collective/c_broadcast_op_npu.cc`
-  `./paddle/fluid/operators/collective/send_v2_op_npu_test.cc`
-  `./paddle/fluid/operators/collective/c_reducescatter_op.cc`
-  `./paddle/fluid/operators/collective/send_v2_op.cc`
-  `./paddle/fluid/operators/collective/c_broadcast_op.cc`
-  `./paddle/fluid/operators/collective/mp_allreduce_sum_op.cc`
-  `./paddle/fluid/operators/collective/partial_allgather_op_npu.cc`
-  `./paddle/fluid/operators/collective/c_allgather_op_npu.cc`
-  `./paddle/fluid/operators/collective/recv_v2_op.cc`
-  `./paddle/fluid/operators/collective/partial_allgather_op.cc`
-  `./paddle/fluid/operators/collective/c_comm_init_hccl_op.cc`
-  `./paddle/fluid/operators/collective/c_reducescatter_op_npu_test.cc`
-  `./paddle/fluid/operators/collective/c_broadcast_op_npu_test.cc`
-  `./paddle/fluid/operators/collective/c_gen_hccl_id_op.cc`
-  `./paddle/fluid/operators/collective/c_allgather_op.cc`
-  `./paddle/fluid/operators/collective/recv_v2_op_npu.cc`
-  `./paddle/fluid/operators/collective/c_sync_comm_stream_op.h`
-  `./paddle/fluid/operators/collective/partial_send_op_npu.cc`
-  `./paddle/fluid/operators/collective/c_allreduce_op.h`
-  `./paddle/fluid/operators/collective/c_reducescatter_op_npu.cc`
-  `./paddle/fluid/operators/collective/gen_hccl_id_op_helper.cc`
-  `./paddle/fluid/operators/collective/partial_recv_op_npu.cc`
-  `./paddle/fluid/operators/collective/c_allreduce_sum_op_npu_test.cc`
-  `./paddle/fluid/operators/collective/recv_v2_op_npu_test.cc`
-  `./paddle/fluid/operators/collective/c_sync_comm_stream_op_npu_test.cc`
-  `./paddle/fluid/operators/collective/send_v2_op_npu.cc`
-  `./paddle/fluid/operators/norm_op.cc`
-  `./paddle/fluid/operators/run_program_op_npu.cc`
-  `./paddle/fluid/operators/tensor_formatter.cc`
-  `./paddle/fluid/operators/math/CMakeLists.txt`
-  `./paddle/fluid/operators/math/concat_and_split.cc`
-  `./paddle/fluid/operators/expand_v2_op.h`
-  `./paddle/fluid/operators/elementwise/CMakeLists.txt`
-  `./paddle/fluid/operators/amp/check_finite_and_unscale_op.cc`
-  `./paddle/fluid/operators/amp/CMakeLists.txt`
-  `./paddle/fluid/operators/unsqueeze_op_npu.cc`
-  `./paddle/fluid/operators/memcpy_op.h`
-  `./paddle/fluid/operators/coalesce_tensor_op.cc`
-  `./paddle/fluid/operators/detection/CMakeLists.txt`
-  `./paddle/fluid/operators/controlflow/while_op_helper.cc`
-  `./paddle/fluid/operators/controlflow/conditional_block_op.h`
-  `./paddle/fluid/operators/softmax_with_cross_entropy_op.cc`
-  `./paddle/fluid/framework/pipeline_trainer.cc`
-  `./paddle/fluid/framework/garbage_collector.cc`
-  `./paddle/fluid/framework/section_worker.cc`
-  `./paddle/fluid/framework/executor.cc`
-  `./paddle/fluid/framework/tensor_util_test.cc`
-  `./paddle/fluid/framework/parallel_executor.cc`
-  `./paddle/fluid/framework/tensor_util.h`
-  `./paddle/fluid/framework/details/CMakeLists.txt`
-  `./paddle/fluid/framework/details/nan_inf_utils_detail.cc`
-  `./paddle/fluid/framework/details/nan_inf_utils.h`
-  `./paddle/fluid/framework/type_defs.h`
-  `./paddle/fluid/framework/phi_utils.cc`
-  `./paddle/fluid/framework/trainer_factory.cc`
-  `./paddle/fluid/framework/tensor_util.cc`
-  `./paddle/fluid/framework/trainer.h`
-  `./paddle/fluid/framework/var_type_traits.h`
-  `./paddle/fluid/framework/tensor_test.cc`
-  `./paddle/fluid/framework/operator.cc`
-  `./paddle/fluid/framework/fleet/CMakeLists.txt`
-  `./paddle/fluid/framework/fleet/ascend_wrapper.h`
-  `./paddle/fluid/framework/fleet/ascend_wrapper.cc`
-  `./paddle/fluid/framework/new_executor/interpreter/execution_config.cc`
-  `./paddle/fluid/framework/new_executor/interpreter/interpreter_util.cc`
-  `./paddle/fluid/framework/new_executor/interpretercore.cc`
-  `./paddle/fluid/framework/device_worker.h`
-  `./paddle/fluid/framework/garbage_collector.h`
-  `./paddle/fluid/framework/device_worker_factory.cc`
-  `./paddle/fluid/distributed/fleet_executor/test/CMakeLists.txt`
-  `./paddle/fluid/distributed/fleet_executor/message_bus.cc`
-  `./paddle/fluid/platform/init.cc`
-  `./paddle/fluid/platform/CMakeLists.txt`
-  `./paddle/fluid/platform/device_context.cc`
-  `./paddle/fluid/platform/device_context.h`
-  `./paddle/fluid/platform/gen_comm_id_helper.cc`
-  `./paddle/fluid/platform/gen_comm_id_helper.h`
-  `./paddle/fluid/platform/device_event_npu.cc`
-  `./paddle/fluid/platform/device/CMakeLists.txt`
-  `./paddle/fluid/platform/device/device_wrapper.h`
-  `./paddle/fluid/platform/device/npu/npu_resource_pool.h`
-  `./paddle/fluid/platform/device/npu/CMakeLists.txt`
-  `./paddle/fluid/platform/device/npu/enforce_npu.h`
-  `./paddle/fluid/platform/device/npu/npu_resource_pool.cc`
-  `./paddle/fluid/platform/device/npu/hccl_helper.h`
-  `./paddle/fluid/platform/device/npu/npu_op_runner.h`
-  `./paddle/fluid/platform/device/npu/npu_info.h`
-  `./paddle/fluid/platform/device/npu/npu_collective_helper.cc`
-  `./paddle/fluid/platform/device/npu/npu_stream.h`
-  `./paddle/fluid/platform/device/npu/ascend_npu_info.h`
-  `./paddle/fluid/platform/device/npu/dynload/CMakeLists.txt`
-  `./paddle/fluid/platform/device/npu/dynload/hccl.h`
-  `./paddle/fluid/platform/device/npu/dynload/hccl.cc`
-  `./paddle/fluid/platform/stream_callback_manager.cc`
-  `./paddle/fluid/platform/place.h`
-  `./paddle/fluid/platform/collective_helper.h`
-  `./paddle/fluid/platform/dynload/CMakeLists.txt`
-  `./paddle/fluid/platform/device_event.h`
-  `./paddle/fluid/inference/analysis/passes/ir_params_sync_among_devices_pass.h`
-  `./paddle/fluid/inference/analysis/passes/ir_params_sync_among_devices_pass.cc`
-  `./paddle/fluid/inference/tests/api/CMakeLists.txt`
-  `./paddle/fluid/inference/api/analysis_predictor.cc`
-  `./paddle/fluid/inference/api/analysis_config.cc`
-  `./paddle/fluid/inference/api/details/zero_copy_tensor_test.cc`
-  `./paddle/fluid/inference/api/details/zero_copy_tensor.cc`
-  `./paddle/fluid/inference/api/api_impl_tester.cc`
-  `./paddle/fluid/inference/api/api_impl.cc`
-  `./paddle/fluid/pybind/CMakeLists.txt`
-  `./paddle/fluid/pybind/tensor_py.h`
-  `./paddle/fluid/pybind/ascend_wrapper_py.h`
-  `./paddle/fluid/pybind/eager_legacy_op_function_generator.cc`
-  `./paddle/fluid/pybind/tensor.cc`
-  `./paddle/fluid/pybind/ascend_wrapper_py.cc`
-  `./paddle/fluid/pybind/parallel_executor.cc`
-  `./paddle/fluid/pybind/inference_api.cc`
-  `./paddle/fluid/pybind/imperative.cc`
-  `./paddle/fluid/pybind/pybind.cc`
-  `./paddle/fluid/pybind/place.cc`
-  `./paddle/testing/paddle_gtest_main.cc`
-  `./paddle/scripts/paddle_build.sh`
-  `./paddle/phi/kernels/funcs/interpolate_function.h`
-  `./paddle/phi/kernels/funcs/strided_memcpy.h`
-  `./paddle/phi/core/utils/visit_place.h`
-  `./paddle/phi/core/flags.cc`
-  `./paddle/phi/backends/device_memory_aligment.h`
-  `./paddle/phi/backends/npu/npu_info.h`
-  `./paddle/phi/backends/context_pool.cc`
-  `./paddle/phi/backends/dynload/CMakeLists.txt`
-  `./paddle/phi/backends/dynload/dynamic_loader.cc`

8. 清理 `is_compiled_with_npu` 相关代码：

- `./tools/get_quick_disable_lt.py`
- `./test/custom_kernel/custom_kernel_dot_c_setup.py`
- `./test/custom_kernel/custom_kernel_dot_setup.py`
- `./python/paddle/nn/functional/conv.py`
- `./python/paddle/distributed/transpiler/collective.py`
- `./python/paddle/distributed/collective.py`
- `./python/paddle/distributed/launch/context/device.py`
- `./python/paddle/distributed/ps/utils/collective_transpiler.py`
- `./python/paddle/distributed/fleet/layers/mpu/mp_ops.py`
- `./python/paddle/distributed/fleet/utils/hybrid_parallel_inference.py`
- `./python/paddle/distributed/fleet/meta_parallel/sharding/group_sharded_utils.py`
- `./python/paddle/distributed/fleet/launch.py`
- `./python/paddle/distributed/fleet/meta_optimizers/sharding/offload_helper.py`
- `./python/paddle/distributed/fleet/meta_optimizers/common.py`
- `./python/paddle/distributed/fleet/meta_optimizers/sharding_optimizer.py`
- `./python/paddle/distributed/fleet/launch_utils.py`
- `./python/paddle/distributed/parallel.py`
- `./python/paddle/hapi/model.py`
- `./python/paddle/__init__.py`
- `./python/paddle/utils/install_check.py`
- `./python/paddle/fluid/dygraph/varbase_patch_methods.py`
- `./python/paddle/fluid/tests/unittests/c_embedding_op_base.py`
- `./python/paddle/fluid/tests/unittests/test_device.py`
- `./python/paddle/fluid/tests/unittests/test_dist_base.py`
- `./python/paddle/fluid/tests/unittests/eager_op_test.py`
- `./python/paddle/fluid/tests/unittests/test_var_base.py`
- `./python/paddle/fluid/framework.py`
- `./python/paddle/fluid/__init__.py`
- `./python/paddle/fluid/device_worker.py`
- `./python/paddle/fluid/optimizer.py`
- `./python/paddle/fluid/executor.py`
- `./python/paddle/amp/grad_scaler.py`
- `./python/paddle/static/nn/common.py`
- `./python/paddle/static/amp/decorator.py`
- `./python/paddle/static/amp/fp16_lists.py`
- `./python/paddle/static/amp/amp_nn.py`
- `./python/paddle/device/__init__.py`
- `./paddle/fluid/pybind/pybind.cc`

9. 清理 `PADDLE_WITH_ASCEND` 相关代码，具体如下：

./paddle/fluid/operators/ascend_trigger_op.h
./paddle/fluid/pybind/tensor.cc
./paddle/fluid/pybind/ascend_wrapper_py.cc
./paddle/fluid/pybind/parallel_executor.cc
./paddle/fluid/pybind/pybind.cc
./paddle/fluid/pybind/place.cc

10. 清理 `WITH_ASCEND` 相关代码，具体如下：

- `./CMakeLists.txt`
- `./test/asp/CMakeLists.txt`
- `./cmake/third_party.cmake`
- `./cmake/configure.cmake`
- `./cmake/external/warpctc.cmake`
- `./cmake/external/protobuf.cmake`
- `./cmake/external/threadpool.cmake`
- `./cmake/external/gloo.cmake`
- `./cmake/external/ascend.cmake`
- `./python/paddle/fluid/tests/unittests/CMakeLists.txt`
- `./python/paddle/fluid/tests/unittests/collective/CMakeLists.txt`
- `./python/paddle/fluid/tests/unittests/collective/fleet/CMakeLists.txt`
- `./python/paddle/fluid/tests/unittests/distributed_passes/CMakeLists.txt`
- `./paddle/fluid/operators/ascend_trigger_op.h`
- `./paddle/fluid/operators/CMakeLists.txt`
- `./paddle/fluid/framework/fleet/CMakeLists.txt`
- `./paddle/fluid/distributed/fleet_executor/test/CMakeLists.txt`
- `./paddle/fluid/platform/device/CMakeLists.txt`
- `./paddle/fluid/platform/device/npu/CMakeLists.txt`
- `./paddle/fluid/inference/tests/api/CMakeLists.txt`
- `./paddle/fluid/pybind/CMakeLists.txt`

11. 清理 paddle_build.sh 脚本中所有的 WITH_MLU WITH_ASCEND* 关键字

- `./paddle/scripts/paddle_build.sh`

KimBioInfoStudio · 2023-03-30T12:20:46Z

任务编号	任务内容	认领人	PR链接
4	清理 `is_compiled_with_mlu` 相关代码	@KimBioInfoStudio	#52378
8	清理 `is_compiled_with_npu` 相关代码	@KimBioInfoStudio	#52385
11	清理 `paddle_build.sh` 脚本中所有的 `WITH_MLU` `WITH_ASCEND*`关键字	@KimBioInfoStudio	#52373
12	清理 WITH_CNCL 相关代码	@KimBioInfoStudio	#52434
14	清理 dockerfile	@KimBioInfoStudio	#52474

遗留问题

代码冲突

rem is_compiled_with_npu #52385 rem is_compiled_with_mlu #52378 之间存在冲突, 任何一个先merge, 都需要大量工作 rebase 后一个
rem is_compiled_with_npu #52385 / rem is_compiled_with_mlu #52378 与其他子任务存在冲突

修改造成功能缺失

distibuted 相关修改 block PaddleCustomDevice distribued 功能，需要为 CustomDevice 重新设计 API 昇腾和寒武纪相关代码退场 Tracking Issue #52319 (comment)
./python/paddle/distributed/fleet/layers/mpu/mp_ops.py 有残留
./python/paddle/distributed/fleet/meta_optimizers/common.py 有残留
./python/paddle/distributed/fleet/meta_optimizers/sharding_optimizer.py 有残留
./python/paddle/distributed/fleet/utils/hybrid_parallel_inference.py
./python/paddle/distributed/fleet/meta_parallel/sharding/group_sharded_utils.py
./python/paddle/distributed/fleet/meta_parallel/sharding/group_sharded_utils.py
./python/paddle/fluid/tests/unittests/c_embedding_op_base.py
./paddle/fluid/pybind/pybind.cc 大概率失去 profiling 功能

暂时不能完全去掉 npu 相关代码

KimBioInfoStudio · 2023-03-30T12:57:22Z

@GreatV distributed 相关的部分代码不能直接删除，需要修改, 目前 PaddleCustomDevice 一些 distributed 相关功能并没有完全 decouple cc @luotao1 @qili93

KimBioInfoStudio · 2023-03-30T12:59:17Z

CC: @KevinYuk @yangulei

KimBioInfoStudio · 2023-03-30T13:56:27Z

seems cncl is mlu ccl, also need clean up @GreatV CC: @luotao1 @qili93

GreatV · 2023-03-30T14:04:00Z

@KimBioInfoStudio 确实缺少了WITH_CNCL的清理

mrcangye · 2023-03-30T14:27:01Z

任务编号	任务内容	认领人	PR链接
1	删除 paddle/fluid/platform/device/mlu 目录	@mrcangye	#52382
9	清理 `PADDLE_WITH_ASCEND` 相关代码	@mrcangye	#52403

Young-Flash · 2023-03-30T15:00:09Z

认领 2

KimBioInfoStudio · 2023-03-30T15:02:29Z

@KimBioInfoStudio 确实缺少了WITH_CNCL的清理

@GreatV 同理，还有 hccl

GreatV · 2023-03-30T15:12:48Z

@KimBioInfoStudio hccl这些应该会在清理 PADDLE_WITH_ASCEND_CL 被清理掉吧

mrcangye · 2023-03-30T15:14:01Z

paddle/fluid/platform/device/mlu/mlu_collective_helper.cc
paddle/fluid/platform/device/mlu/cncl_helper.h
paddle/fluid/platform/device/mlu/device_context_allocator.h
paddle/fluid/platform/device/mlu/device_context_test.cc

第一项任务中，这四个文件是不是也要删除

GreatV · 2023-03-30T15:18:57Z

paddle/fluid/platform/device/mlu/mlu_collective_helper.cc
paddle/fluid/platform/device/mlu/cncl_helper.h
paddle/fluid/platform/device/mlu/device_context_allocator.h
paddle/fluid/platform/device/mlu/mlu_collective_helper.cc

第一项任务中，这四个文件是不是也要删除

应该是的

jjyaoao · 2023-03-30T15:41:09Z

任务编号	任务内容	认领人	PR链接
5	删除 `paddle/fluid/operators/_npu.`、清理 `WITH_ASCEND_INT64` 相关代码	@jjyaoao	#52559
6	删除 `/paddle/fluid/platform/device/npu` 目录	@jjyaoao	#52384

engineer1109 · 2023-03-30T16:03:12Z

任务编号	任务内容	认领人	PR链接
7	清理 WITH_ASCEND_CL PADDLE_WITH_ASCEND_CL WITH_ASCEND_CXX11 相关代码	@engineer	#52448
--	--	--

GreatV · 2023-03-30T16:23:03Z

任务编号	任务内容	认领人	PR链接
3	清理 `WITH_MLU` 相关代码	@GreatV	#52546

KimBioInfoStudio · 2023-03-30T16:30:36Z

paddle/fluid/platform/device/mlu/mlu_collective_helper.cc
paddle/fluid/platform/device/mlu/cncl_helper.h
paddle/fluid/platform/device/mlu/device_context_allocator.h
paddle/fluid/platform/device/mlu/device_context_test.cc

第一项任务中，这四个文件是不是也要删除

是的，现在 mlu 走的是 decvice / custom device, 所以 mlu 整个 dir 都可以删除

KimBioInfoStudio · 2023-03-30T16:32:07Z

@KimBioInfoStudio hccl这些应该会在清理 PADDLE_WITH_ASCEND_CL 被清理掉吧

应该不行，有些代码是在 Python 里面，WITH_ASCEND_CL 大部分都是在 cmake 或者 cc 里面

KimBioInfoStudio · 2023-04-24T07:13:06Z

@ronny1996 @qili93 seems we should remove cnpai for mlu profiler, add redesign a new api for all custom_device cc @luotao1

KimBioInfoStudio · 2023-04-24T08:38:39Z

@luotao1 @Liyulingyue in latest develop adc2b7459f there are still 3 mlu keyword in code

tools/infer_prune_patches/op_registry.h.patch:97:   REGISTER_OP_KERNEL(op_type, MLU, ::paddle::platform::MLUPlace, __VA_ARGS__)
tools/infer_prune_patches/op_registry.h.patch:131: #define REGISTER_OP_MLU_KERNEL_FUNCTOR(op_type, ...)                  \
tools/infer_prune_patches/op_registry.h.patch:133:       op_type, MLU, ::paddle::platform::MLUPlace, DEFAULT_TYPE,       \

luotao1 · 2023-04-24T08:56:08Z

in latest develop adc2b74 there are still 3 mlu keyword in code

@KimBioInfoStudio @Liyulingyue 这个patch先不用改，是推理侧为了裁剪库体积写的脚本，不是简单删除即可。

KimBioInfoStudio · 2023-04-25T04:24:37Z

in latest develop adc2b74 there are still 3 mlu keyword in code

@KimBioInfoStudio @Liyulingyue 这个patch先不用改，是推理侧为了裁剪库体积写的脚本，不是简单删除即可。

Okay, 这个应该是需要重新取diff了，之前一致有个疑惑，删除custom device 代码会导致infer lib size 增加，现在大致猜测是因为 diff apply 不上去导致的

luotao1 · 2023-04-28T06:43:39Z

https://github.com/PaddlePaddle/Paddle/search?q=npu 还剩下最后一项【清理 NPU 相关代码】，还有不少npu代码。

luotao1 · 2023-05-08T07:11:58Z

@KimBioInfoStudio @Liyulingyue 整个 tools/infer_prune_patches 目录都可以删除，#53193 (comment) 已经使用另一种方式在裁剪预测库了。

KimBioInfoStudio · 2023-05-15T06:20:55Z

CMD	`b428e8f35c`
`grep -rni npu \| grep -vi -P "input\|onpush\|onpull\|unpub"`	255
`grep -rni mlu`	0
`grep -rni hccl`	0
`grep -rni ascend\|grep -vi "ascending"`	23
`grep -rni cncl`	0

luotao1 · 2023-05-16T06:41:47Z

感谢 @KimBioInfoStudio 统计

grep -rni ascend|grep -vi "ascending"

ascend 关键字已检查完毕，剩余的都不需要删除

grep -rni npu | grep -vi -P "input|onpush|onpull|unpub"

昇腾和寒武纪相关代码退场 npu相关代码退场3 #53699 已合入，会再次减少
昇腾和寒武纪相关代码退场 npu相关代码退场3 #53699 (comment) 还剩余tools路径和 cmake路径下确认下是不是 npu 的 dockerfile 以及 npu.cmake 等文件，这些都是可以删除的。 @Liyulingyue

luotao1 · 2023-05-18T02:59:13Z

昇腾和寒武纪相关代码退场已全部完成，感谢参与的小伙伴们！

感谢 @GreatV 拆分任务形成 Tracking issue； @KimBioInfoStudio 多次统计需要清理的内容； @Liyulingyue 完成最后的收尾工作，以及所有的小伙伴们！

按 merge 的时间顺序，排名不分先后： @KimBioInfoStudio (5) @mrcangye (2) @jjyaoao (3) @Young-Flash (1) @Liyulingyue (3) @engineer1109 (1) @GreatV (2)

欢迎继续参与快乐开源的其他任务！

GreatV added status/new-issue 新建 type/others 其他问题 labels Mar 29, 2023

paddle-bot bot assigned AnnaTrainingG Mar 29, 2023

paddle-bot bot added the PFCC Paddle Framework Contributor Club，https://github.com/PaddlePaddle/community/tree/master/pfcc label Mar 29, 2023

luotao1 assigned luotao1, qili93 and GreatV and unassigned AnnaTrainingG Mar 30, 2023

KimBioInfoStudio mentioned this issue Mar 30, 2023

rem custom device mlu/npu from build sh #52373

Merged

KimBioInfoStudio mentioned this issue Mar 30, 2023

rem is_compiled_with_mlu #52378

Merged

mrcangye mentioned this issue Mar 30, 2023

【昇腾和寒武纪相关代码退场】Task01 删除 paddle/fluid/platform/device/mlu 目录 #52382

Merged

jjyaoao mentioned this issue Mar 30, 2023

Delete the /paddle/fluid/platform/device/npu directory #52384

Merged

KimBioInfoStudio mentioned this issue Mar 30, 2023

rem is_compiled_with_npu #52385

Merged

KimBioInfoStudio mentioned this issue Apr 19, 2023

rem hccl keywords #53058

Merged

Liyulingyue mentioned this issue Apr 19, 2023

rm is_npu_place #53105

Merged

jjyaoao mentioned this issue Apr 20, 2023

remove is_npu_place and is_npu_pinned_place #53120

Closed

Liyulingyue mentioned this issue Apr 22, 2023

rm mlu #53194

Merged

jjyaoao mentioned this issue Apr 23, 2023

remove is_npu_pinned_place #53212

Closed

jjyaoao mentioned this issue Apr 27, 2023

remove is_npu_pinned_place #53391

Merged

jjyaoao mentioned this issue Apr 28, 2023

remove a part of NPU #53457

Closed

jinyouzhi mentioned this issue Apr 28, 2023

remove NPUPlace/NPUPinnerPlace #53459

Closed

KimBioInfoStudio mentioned this issue Apr 28, 2023

rem npu in test #53469

Merged

This was referenced May 6, 2023

昇腾和寒武纪相关代码退场 npu相关代码退场 #53566

Merged

昇腾和寒武纪相关代码退场 npu相关代码退场2 #53568

Merged

KimBioInfoStudio mentioned this issue May 8, 2023

rem tools/infer_prune_patches #53596

Merged

jjyaoao mentioned this issue May 10, 2023

remove a part of npu #53677

Merged

Liyulingyue mentioned this issue May 10, 2023

昇腾和寒武纪相关代码退场 npu相关代码退场3 #53699

Merged

This was referenced May 17, 2023

rm cmake npu #53869

Merged

rm tools npu #53870

Merged

luotao1 closed this as completed May 18, 2023

paddle-bot bot added status/close 已关闭 and removed status/new-issue 新建 labels May 18, 2023

ziyanwangin mentioned this issue May 30, 2023

Profiler TracerEventType lacks runtime type for custom device #54200

Closed

ccsuzzh mentioned this issue Jun 7, 2023

【第三方库离线编译】Delete concurrentqueue cmake #54438

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

昇腾和寒武纪相关代码退场 Tracking Issue #52319

昇腾和寒武纪相关代码退场 Tracking Issue #52319

GreatV commented Mar 29, 2023 •

edited by luotao1

Loading

GreatV commented Mar 30, 2023 •

edited by luotao1

Loading

KimBioInfoStudio commented Mar 30, 2023 •

edited

Loading

KimBioInfoStudio commented Mar 30, 2023

KimBioInfoStudio commented Mar 30, 2023

KimBioInfoStudio commented Mar 30, 2023

GreatV commented Mar 30, 2023

mrcangye commented Mar 30, 2023 •

edited

Loading

Young-Flash commented Mar 30, 2023

KimBioInfoStudio commented Mar 30, 2023 •

edited

Loading

GreatV commented Mar 30, 2023

mrcangye commented Mar 30, 2023 •

edited

Loading

GreatV commented Mar 30, 2023

jjyaoao commented Mar 30, 2023 •

edited

Loading

engineer1109 commented Mar 30, 2023 •

edited

Loading

GreatV commented Mar 30, 2023 •

edited

Loading

KimBioInfoStudio commented Mar 30, 2023

KimBioInfoStudio commented Mar 30, 2023

KimBioInfoStudio commented Apr 24, 2023 •

edited

Loading

KimBioInfoStudio commented Apr 24, 2023

luotao1 commented Apr 24, 2023

KimBioInfoStudio commented Apr 25, 2023

luotao1 commented Apr 28, 2023

luotao1 commented May 8, 2023

KimBioInfoStudio commented May 15, 2023

luotao1 commented May 16, 2023

luotao1 commented May 18, 2023

昇腾和寒武纪相关代码退场 Tracking Issue #52319

昇腾和寒武纪相关代码退场 Tracking Issue #52319

Comments

GreatV commented Mar 29, 2023 • edited by luotao1 Loading

GreatV commented Mar 30, 2023 • edited by luotao1 Loading

KimBioInfoStudio commented Mar 30, 2023 • edited Loading

遗留问题

代码冲突

修改造成功能缺失

暂时不能完全去掉 npu 相关代码

KimBioInfoStudio commented Mar 30, 2023

KimBioInfoStudio commented Mar 30, 2023

KimBioInfoStudio commented Mar 30, 2023

GreatV commented Mar 30, 2023

mrcangye commented Mar 30, 2023 • edited Loading

Young-Flash commented Mar 30, 2023

KimBioInfoStudio commented Mar 30, 2023 • edited Loading

GreatV commented Mar 30, 2023

mrcangye commented Mar 30, 2023 • edited Loading

GreatV commented Mar 30, 2023

jjyaoao commented Mar 30, 2023 • edited Loading

engineer1109 commented Mar 30, 2023 • edited Loading

GreatV commented Mar 30, 2023 • edited Loading

KimBioInfoStudio commented Mar 30, 2023

KimBioInfoStudio commented Mar 30, 2023

KimBioInfoStudio commented Apr 24, 2023 • edited Loading

KimBioInfoStudio commented Apr 24, 2023

luotao1 commented Apr 24, 2023

KimBioInfoStudio commented Apr 25, 2023

luotao1 commented Apr 28, 2023

luotao1 commented May 8, 2023

KimBioInfoStudio commented May 15, 2023

luotao1 commented May 16, 2023

luotao1 commented May 18, 2023

GreatV commented Mar 29, 2023 •

edited by luotao1

Loading

GreatV commented Mar 30, 2023 •

edited by luotao1

Loading

KimBioInfoStudio commented Mar 30, 2023 •

edited

Loading

mrcangye commented Mar 30, 2023 •

edited

Loading

KimBioInfoStudio commented Mar 30, 2023 •

edited

Loading

mrcangye commented Mar 30, 2023 •

edited

Loading

jjyaoao commented Mar 30, 2023 •

edited

Loading

engineer1109 commented Mar 30, 2023 •

edited

Loading

GreatV commented Mar 30, 2023 •

edited

Loading

KimBioInfoStudio commented Apr 24, 2023 •

edited

Loading