[Feature] Support ball_query with cambricon MLU backend and mlu-ops library. #2520

DanieeelLiu · 2022-12-30T10:25:01Z

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

The motivation of the PR is to support running ball_query with Cambricon MLU backend.

It includes five parts:

1. Add mlu_common_desc.cpp  as mlu-ops adaptor.
2. Add mlu_common_desc.h as part of mlu-ops adaptor.
3. Add ball_query_mlu.cpp to support launching mlu-ops operator in PyTorch
4. Refactor test_ball_query.py to support testing Ball_query_forward with MLU backend.
5. Refactor setup.py to support compile mmcv project with mlu-ops to support MLU backend.

Modification

mlu-ops adaption
Add cpp code of mlu-ops adaption in directory mmcv/ops/csrc/pytorch/mlu/mlu_common_desc.cpp and mmcv/ops/csrc/pytorch/mlu/mlu_common_desc.h.
PyTorch adaptation
Adapt ball_query_forward for PyTorch in mmcv/ops/csrc/pytorch/mlu/ball_query_mlu.cpp.
Unit test
Support test Ball_query with various backends in tests/test_ops/test_ball_query.py.
Compilation
Support compile mmcv with mlu-ops to support MLU backend in setup.py.
Please briefly describe what modification is made in this PR.

BC-breaking (Optional)

Does the modification introduce changes that break the backward-compatibility of the downstream repositories?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.

Use cases (Optional)

If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.

Checklist

Before PR:

I have read and followed the workflow indicated in the CONTRIBUTING.md to create this PR.
Pre-commit or linting tools indicated in CONTRIBUTING.md are used to fix the potential lint issues.
Bug fixes are covered by unit tests, the case that causes the bug should be added in the unit tests.
New functionalities are covered by complete unit tests. If not, please add more unit test to ensure the correctness.
The documentation has been modified accordingly, including docstring or example tutorials.

After PR:

If the modification has potential influence on downstream or other related projects, this PR should be tested with some of those projects, like MMDet or MMCls.
CLA has been signed and all committers have signed the CLA in this PR.

…ibrary.

mmcv/ops/csrc/pytorch/mlu/mlu_common_desc.cpp

mmcv/ops/csrc/pytorch/mlu/ball_query_mlu.cpp

setup.py

tests/test_ops/test_ball_query.py

grimoire · 2023-01-05T04:01:15Z

Test failed with undefined symbol: __bangPopCallConfiguration. Do I need to add some libraries to the environment?

grimoire

LGTM

setup.py

…ibrary. (open-mmlab#2520) * [Feature] Support ball_query with cambricon MLU backend and mlu-ops library. * [Fix] update operator data layout setting. * [Fix] add cxx compile option to avoid symbol conflict. * [Fix] fix lint errors. * [Fix] update ops.md with info of ball_query support by MLU backend. * [Feature] Fix typo. * [Fix] Remove print. * [Fix] get mlu-ops from MMCV_MLU_OPS_PATH env. * [Fix] update MMCV_MLU_OPS_PATH check logic. * [Fix] update error info when failed to download mlu-ops. * [Fix] check mlu-ops version matching info in mmcv. * [Fix] revise wrong filename. * [Fix] remove f.close and re.

* [Feature] Support Voxelization with cambricon MLU device (#2500) * [Feature] Support hard_voxelize with cambricon MLU backend * [Feature](bangc-ops): add voxelization op * [Feature](bangc-ops): add voxelization op * [Feature](bangc-ops): add voxelization op * [Feature](bangc-ops): add voxelization op * [Feature](bangc-ops): add voxelization op * [Feature](bangc-ops): add voxelization op * [Feature](bangc-ops): add voxelization op * [Feature](bangc-ops): add voxelization op * [Enhance] Optimize the performace of ms_deform_attn for MLU device (#2510) * ms_opt * ms_opt * ms_opt * ms_opt * ms_opt * [Feature] ms_deform_attn performance optimization * [Feature] ms_deform_attn performance optimization * [Feature] ms_deform_attn performance optimization * [Feature] Support ball_query with cambricon MLU backend and mlu-ops library. (#2520) * [Feature] Support ball_query with cambricon MLU backend and mlu-ops library. * [Fix] update operator data layout setting. * [Fix] add cxx compile option to avoid symbol conflict. * [Fix] fix lint errors. * [Fix] update ops.md with info of ball_query support by MLU backend. * [Feature] Fix typo. * [Fix] Remove print. * [Fix] get mlu-ops from MMCV_MLU_OPS_PATH env. * [Fix] update MMCV_MLU_OPS_PATH check logic. * [Fix] update error info when failed to download mlu-ops. * [Fix] check mlu-ops version matching info in mmcv. * [Fix] revise wrong filename. * [Fix] remove f.close and re. * [Docs] Steps to compile mmcv-full on MLU machine (#2571) * [Docs] Steps to compile mmcv-full on MLU machine * [Docs] Adjust paragraph order * Update docs/zh_cn/get_started/build.md Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com> * Update docs/zh_cn/get_started/build.md Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com> * Update docs/en/get_started/build.md Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com> * Update docs/en/get_started/build.md Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com> * [Docs] Modify the format --------- Co-authored-by: budefei <budefei@cambricon.com> Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com> * [Fix] Fix tensor descriptor setting in MLU ball_query. (#2579) * [Feature] Add MLU support for Sparse Convolution op (#2589) * [Feature] Add sparse convolution MLU API * [Feature] update cpp code style * end-of-file * delete libext.a * code style * update ops.md --------- Co-authored-by: budefei <budefei@cambricon.com> * [Enhancement] Replace the implementation of deform_roi_pool with mlu-ops (#2598) * [Feature] Replace the implementation of deform_roi_pool with mlu-ops * [Feature] Modify code --------- Co-authored-by: budefei <budefei@cambricon.com> * [Enhancement] ms_deform_attn performance optimization (#2616) * ms_opt_v2 * ms_opt_v2_1 * optimize MultiScaleDeformableAttention ops for MLU * ms_opt_v2_1 * [Feature] ms_deform_attn performance optimization V2 * [Feature] ms_deform_attn performance optimization V2 * [Feature] ms_deform_attn performance optimization V2 * [Feature] ms_deform_attn performance optimization V2 * [Feature] ms_deform_attn performance optimization V2 * [Feature] ms_deform_attn performance optimization V2 * [Feature] ms_deform_attn performance optimization V2 --------- Co-authored-by: dongchengwei <dongchengwei@cambricon.com> * [Feature] Support NmsRotated with cambricon MLU backend (#2643) * [Feature] Support NmsRotated with cambricon MLU backend * [Feature] remove foolproofs in nms_rotated_mlu.cpp * [Feature] fix lint in test_nms_rotated.py * [Feature] fix kMLU not found in nms_rotated.cpp * [Feature] modify mlu support in nms.py * [Feature] modify nms_rotated support in ops.md * [Feature] modify ops/nms.py * [Enhance] Add a default value for MMCV_MLU_ARGS (#2688) * add mlu_args * add mlu_args * Modify the code --------- Co-authored-by: budefei <budefei@cambricon.com> * [Enhance] Ignore mlu-ops files (#2691) Co-authored-by: budefei <budefei@cambricon.com> --------- Co-authored-by: ZShaopeng <108382403+ZShaopeng@users.noreply.github.com> Co-authored-by: BinZheng <38182684+Wickyzheng@users.noreply.github.com> Co-authored-by: liuduanhui <103939338+DanieeelLiu@users.noreply.github.com> Co-authored-by: budefei <budefei@cambricon.com> Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com> Co-authored-by: duzekun <108381389+duzekunKTH@users.noreply.github.com> Co-authored-by: dongchengwei <dongchengwei@cambricon.com> Co-authored-by: liuyuan1-v <125547457+liuyuan1-v@users.noreply.github.com>

…ibrary. (open-mmlab#2520) * [Feature] Support ball_query with cambricon MLU backend and mlu-ops library. * [Fix] update operator data layout setting. * [Fix] add cxx compile option to avoid symbol conflict. * [Fix] fix lint errors. * [Fix] update ops.md with info of ball_query support by MLU backend. * [Feature] Fix typo. * [Fix] Remove print. * [Fix] get mlu-ops from MMCV_MLU_OPS_PATH env. * [Fix] update MMCV_MLU_OPS_PATH check logic. * [Fix] update error info when failed to download mlu-ops. * [Fix] check mlu-ops version matching info in mmcv. * [Fix] revise wrong filename. * [Fix] remove f.close and re.

* [Feature] Support Voxelization with cambricon MLU device (open-mmlab#2500) * [Feature] Support hard_voxelize with cambricon MLU backend * [Feature](bangc-ops): add voxelization op * [Feature](bangc-ops): add voxelization op * [Feature](bangc-ops): add voxelization op * [Feature](bangc-ops): add voxelization op * [Feature](bangc-ops): add voxelization op * [Feature](bangc-ops): add voxelization op * [Feature](bangc-ops): add voxelization op * [Feature](bangc-ops): add voxelization op * [Enhance] Optimize the performace of ms_deform_attn for MLU device (open-mmlab#2510) * ms_opt * ms_opt * ms_opt * ms_opt * ms_opt * [Feature] ms_deform_attn performance optimization * [Feature] ms_deform_attn performance optimization * [Feature] ms_deform_attn performance optimization * [Feature] Support ball_query with cambricon MLU backend and mlu-ops library. (open-mmlab#2520) * [Feature] Support ball_query with cambricon MLU backend and mlu-ops library. * [Fix] update operator data layout setting. * [Fix] add cxx compile option to avoid symbol conflict. * [Fix] fix lint errors. * [Fix] update ops.md with info of ball_query support by MLU backend. * [Feature] Fix typo. * [Fix] Remove print. * [Fix] get mlu-ops from MMCV_MLU_OPS_PATH env. * [Fix] update MMCV_MLU_OPS_PATH check logic. * [Fix] update error info when failed to download mlu-ops. * [Fix] check mlu-ops version matching info in mmcv. * [Fix] revise wrong filename. * [Fix] remove f.close and re. * [Docs] Steps to compile mmcv-full on MLU machine (open-mmlab#2571) * [Docs] Steps to compile mmcv-full on MLU machine * [Docs] Adjust paragraph order * Update docs/zh_cn/get_started/build.md Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com> * Update docs/zh_cn/get_started/build.md Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com> * Update docs/en/get_started/build.md Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com> * Update docs/en/get_started/build.md Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com> * [Docs] Modify the format --------- Co-authored-by: budefei <budefei@cambricon.com> Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com> * [Fix] Fix tensor descriptor setting in MLU ball_query. (open-mmlab#2579) * [Feature] Add MLU support for Sparse Convolution op (open-mmlab#2589) * [Feature] Add sparse convolution MLU API * [Feature] update cpp code style * end-of-file * delete libext.a * code style * update ops.md --------- Co-authored-by: budefei <budefei@cambricon.com> * [Enhancement] Replace the implementation of deform_roi_pool with mlu-ops (open-mmlab#2598) * [Feature] Replace the implementation of deform_roi_pool with mlu-ops * [Feature] Modify code --------- Co-authored-by: budefei <budefei@cambricon.com> * [Enhancement] ms_deform_attn performance optimization (open-mmlab#2616) * ms_opt_v2 * ms_opt_v2_1 * optimize MultiScaleDeformableAttention ops for MLU * ms_opt_v2_1 * [Feature] ms_deform_attn performance optimization V2 * [Feature] ms_deform_attn performance optimization V2 * [Feature] ms_deform_attn performance optimization V2 * [Feature] ms_deform_attn performance optimization V2 * [Feature] ms_deform_attn performance optimization V2 * [Feature] ms_deform_attn performance optimization V2 * [Feature] ms_deform_attn performance optimization V2 --------- Co-authored-by: dongchengwei <dongchengwei@cambricon.com> * [Feature] Support NmsRotated with cambricon MLU backend (open-mmlab#2643) * [Feature] Support NmsRotated with cambricon MLU backend * [Feature] remove foolproofs in nms_rotated_mlu.cpp * [Feature] fix lint in test_nms_rotated.py * [Feature] fix kMLU not found in nms_rotated.cpp * [Feature] modify mlu support in nms.py * [Feature] modify nms_rotated support in ops.md * [Feature] modify ops/nms.py * [Enhance] Add a default value for MMCV_MLU_ARGS (open-mmlab#2688) * add mlu_args * add mlu_args * Modify the code --------- Co-authored-by: budefei <budefei@cambricon.com> * [Enhance] Ignore mlu-ops files (open-mmlab#2691) Co-authored-by: budefei <budefei@cambricon.com> --------- Co-authored-by: ZShaopeng <108382403+ZShaopeng@users.noreply.github.com> Co-authored-by: BinZheng <38182684+Wickyzheng@users.noreply.github.com> Co-authored-by: liuduanhui <103939338+DanieeelLiu@users.noreply.github.com> Co-authored-by: budefei <budefei@cambricon.com> Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com> Co-authored-by: duzekun <108381389+duzekunKTH@users.noreply.github.com> Co-authored-by: dongchengwei <dongchengwei@cambricon.com> Co-authored-by: liuyuan1-v <125547457+liuyuan1-v@users.noreply.github.com>

…ibrary. (open-mmlab#2520) * [Feature] Support ball_query with cambricon MLU backend and mlu-ops library. * [Fix] update operator data layout setting. * [Fix] add cxx compile option to avoid symbol conflict. * [Fix] fix lint errors. * [Fix] update ops.md with info of ball_query support by MLU backend. * [Feature] Fix typo. * [Fix] Remove print. * [Fix] get mlu-ops from MMCV_MLU_OPS_PATH env. * [Fix] update MMCV_MLU_OPS_PATH check logic. * [Fix] update error info when failed to download mlu-ops. * [Fix] check mlu-ops version matching info in mmcv. * [Fix] revise wrong filename. * [Fix] remove f.close and re.

[Feature] Support ball_query with cambricon MLU backend and mlu-ops l…

ae10c55

…ibrary.

mm-assistant bot assigned ice-tong Dec 30, 2022

Qiza-lyhm reviewed Dec 31, 2022

View reviewed changes

mmcv/ops/csrc/pytorch/mlu/mlu_common_desc.cpp Outdated Show resolved Hide resolved

Qiza-lyhm reviewed Dec 31, 2022

View reviewed changes

mmcv/ops/csrc/pytorch/mlu/ball_query_mlu.cpp Show resolved Hide resolved

[Fix] update operator data layout setting.

0e792db

Qiza-lyhm approved these changes Jan 3, 2023

View reviewed changes

Qiza-lyhm reviewed Jan 3, 2023

View reviewed changes

setup.py Show resolved Hide resolved

DanieeelLiu added 2 commits January 4, 2023 09:58

[Fix] add cxx compile option to avoid symbol conflict.

28c332e

[Fix] fix lint errors.

fd09709

zhouzaida added the MLU label Jan 4, 2023

[Fix] update ops.md with info of ball_query support by MLU backend.

7362795

zhouzaida requested a review from grimoire January 4, 2023 08:53

grimoire reviewed Jan 5, 2023

View reviewed changes

setup.py Outdated Show resolved Hide resolved

[Feature] Fix typo.

c063239

grimoire reviewed Jan 5, 2023

View reviewed changes

tests/test_ops/test_ball_query.py Outdated Show resolved Hide resolved

[Fix] Remove print.

9ca1155

grimoire approved these changes Jan 6, 2023

View reviewed changes

DanieeelLiu added 2 commits January 12, 2023 18:10

[Fix] get mlu-ops from MMCV_MLU_OPS_PATH env.

436aa1c

[Fix] update MMCV_MLU_OPS_PATH check logic.

7a1df66

zhouzaida reviewed Jan 13, 2023

View reviewed changes

setup.py Outdated Show resolved Hide resolved

[Fix] update error info when failed to download mlu-ops.

fe4b56b

zhouzaida approved these changes Jan 13, 2023

View reviewed changes

[Fix] check mlu-ops version matching info in mmcv.

b8211a2

PetrelYy approved these changes Jan 16, 2023

View reviewed changes

defei-coder reviewed Jan 16, 2023

View reviewed changes

setup.py Outdated Show resolved Hide resolved

setup.py Outdated Show resolved Hide resolved

setup.py Outdated Show resolved Hide resolved

[Fix] revise wrong filename.

9835891

DanieeelLiu commented Jan 17, 2023

View reviewed changes

setup.py Outdated Show resolved Hide resolved

zhouzaida reviewed Jan 17, 2023

View reviewed changes

setup.py Outdated Show resolved Hide resolved

zhouzaida reviewed Jan 17, 2023

View reviewed changes

setup.py Outdated Show resolved Hide resolved

[Fix] remove f.close and re.

219fdee

zhouzaida approved these changes Jan 18, 2023

View reviewed changes

zhouzaida merged commit dfb0380 into open-mmlab:master Jan 18, 2023

defei-coder mentioned this pull request Mar 27, 2023

Pick MLU modifications from master (1.x) to main (2.x) #2704

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Support ball_query with cambricon MLU backend and mlu-ops library. #2520

[Feature] Support ball_query with cambricon MLU backend and mlu-ops library. #2520

DanieeelLiu commented Dec 30, 2022 •

edited

Loading

grimoire commented Jan 5, 2023

grimoire left a comment

[Feature] Support ball_query with cambricon MLU backend and mlu-ops library. #2520

[Feature] Support ball_query with cambricon MLU backend and mlu-ops library. #2520

Conversation

DanieeelLiu commented Dec 30, 2022 • edited Loading

Motivation

Modification

BC-breaking (Optional)

Use cases (Optional)

Checklist

grimoire commented Jan 5, 2023

grimoire left a comment

Choose a reason for hiding this comment

DanieeelLiu commented Dec 30, 2022 •

edited

Loading