[WIP][Feature] Support TransFusion #2547

wep21 · 2023-05-23T07:32:13Z

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

Please describe the motivation of this PR and the goal you want to achieve through this PR.

import TransFusion as project

Modification

Please briefly describe what modification is made in this PR.
add transfusion implementation

BC-breaking (Optional)

Does the modification introduce changes that break the back-compatibility of the downstream repos?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.

Use cases (Optional)

If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.

Checklist

Pre-commit or other linting tools are used to fix the potential lint issues.
The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
If the modification has potential influence on downstream projects, this PR should be tested with downstream projects.
The documentation has been modified accordingly, like docstring or example tutorials.

CLAassistant · 2023-05-23T07:32:18Z

All committers have signed the CLA.

JingweiZhang12 · 2023-05-25T02:59:49Z

@wep21 Thanks for your awesome contribution, and we're excited to invite you to join core developer community of mmdet3d. Just add our assistant using the WeChat ID: openmmlabwx. When sending the friend request, remember to include the remark "mmsig + Github ID". We will invite you to MMDet3D developer group.

codecov · 2023-05-25T03:02:52Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (0ef13b8) 47.26% compared to head (2587d4f) 47.26%.

Additional details and impacted files

@@           Coverage Diff            @@
##           dev-1.x    #2547   +/-   ##
========================================
  Coverage    47.26%   47.26%           
========================================
  Files          277      277           
  Lines        23415    23415           
  Branches      3655     3655           
========================================
  Hits         11067    11067           
  Misses       11635    11635           
  Partials       713      713

Flag	Coverage Δ
unittests	`47.26% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

projects/TransFusion/configs/transfusion_nusc_pillar_L.py

JingweiZhang12 · 2023-05-25T03:08:43Z

projects/TransFusion/transfusion/hungarian_assigner.py

+
+
+@TASK_UTILS.register_module()
+class HeuristicAssigner3D(BaseAssigner):


Could reuse the implementation in BEVFusion and del this file?

from ..BEVFusion.bevfusion.utils import HungarianAssigner3D from ..BEVFusion.bevfusion.transformer import TransformerDecoderLayer # noqa F401 from ..BEVFusion.bevfusion.utils import TransFusionBBoxCoder

Sure. Is it better to merge these changes?

@JingweiZhang12 I pushed the code based on the original repository. I'm currently working on using BEVFusion implementation here, but loss_bbox doesn't decrease compared to the original one.
I guess I made a mistake when integrating your code, so I appreciate it if you could give me some advice to resolve this issue.

@wep21 Hi, the original BEVFusion has a potential bug: mit-han-lab/bevfusion#419. This will cause the loss_bbox can not decrease when both using lidar and camera modalities. We have fixed it in the latest branch. Also, the BEVfusion has been merged. You can pull the latest dev-1.x branch.

@JingweiZhang12 I found out the changes from original code cause problem when training transfusion. I reverted the changes here and then loss_bbox and matched ious become sane.
Do you have any thoughts about this?

Hi， this is because the voxelization output order of the BEVFusion is different from ours. BEVFusion: xyz, Ours: zyx. Here is easily manifested: https://github.com/open-mmlab/mmdetection3d/blob/main/projects/BEVFusion/bevfusion/sparse_encoder.py#L15

I see. Is it better to adapt the pillar/voxelization output of the TransFusion to the coordinate of BEVFusion?

It may not have an influence on the accuracy of the model. The style of the original Transfusion is the same as ours.

@JingweiZhang12 I would like to use the same coordinate as original, so is it better to create another head implementation in transfusion?

@wep21 Yep, it's better to use the original head implements of TransFusioin to align the training precision easily since the BEVFusion head may modify something and have some potential incompatibilities.

projects/TransFusion/transfusion/transfusion_bbox_corder.py

JingweiZhang12 · 2023-05-25T03:19:03Z

Could you update your progress of implementation of TransFusion according to the bellow items:

JingweiZhang12 · 2023-06-06T07:39:11Z

projects/TransFusion/transfusion/transfusion.py

+
+@MODELS.register_module()
+class TransFusionDetector(MVXTwoStageDetector):
+    """Base class of Multi-modality VoxelNet."""


Please update this docstring.

updated at fd47bf7

JingweiZhang12 · 2023-06-06T07:40:50Z

projects/BEVFusion/bevfusion/transfusion_head.py

                # NOTE: fix
-                draw_heatmap_gaussian(heatmap[gt_labels_3d[idx]],
-                                      center_int[[1, 0]], radius)
+                # draw_heatmap_gaussian(heatmap[gt_labels_3d[idx]],


Please remove lines 729-731 since it's unnecessary.

remove these lines in new transfusion head at bb0a95e.
Also, I renamed head in bevfusion due to name conflict at dc948cc. Please let me know if this change is not applicable.

JingweiZhang12 · 2023-06-06T07:44:47Z

projects/TransFusion/configs/transfusion_nusc_pillar_L.py

+_base_ = ['../../../configs/_base_/default_runtime.py']
+custom_imports = dict(
+    imports=['projects.TransFusion.transfusion'], allow_failed_imports=False)
+point_cloud_range = [-51.2, -51.2, -5.0, 51.2, 51.2, 3.0]


Please rename this file as transfusion_lidar_pillar02_second_secfpn_8xb2-cyclic-20e_nus-3d.py

renamed config at 615cb1d.

JingweiZhang12 · 2023-06-06T07:48:37Z

projects/TransFusion/configs/transfusion_nusc_pillar_L.py

+backend_args = None
+
+model = dict(
+    type='TransFusionDetector',


Suggested change

type='TransFusionDetector',

type='TransFusion',

Remind to modify the corresponding name.

rename detector class name at 5608d19

JingweiZhang12 · 2023-06-06T07:54:46Z

@wep21 Hi, have you aligned the inference precision of TransFusion(only LiDAR modality)?

wep21 · 2023-06-06T18:56:15Z

@JingweiZhang12

Hi, have you aligned the inference precision of TransFusion(only LiDAR modality)?

I'm sorry, but not yet. I've tested with only nuscenes-mini. Currently, I don't have enough gpu environment, it may take a time to train and test with nuscenes full dataset.

JingweiZhang12 · 2023-06-07T09:13:32Z

@JingweiZhang12

Hi, have you aligned the inference precision of TransFusion(only LiDAR modality)?

I'm sorry, but not yet. I've tested with only nuscenes-mini. Currently, I don't have enough gpu environment, it may take a time to train and test with nuscenes full dataset.

That's OK. Suggest you can make sure the results of one forward are the same as that of the official implementation firstly.

sunjiahao1999 · 2023-08-16T15:22:16Z

@wep21 Hi, any progress on TransFusion so far?

Signed-off-by: wep21 <daisuke.nishimatsu1021@gmail.com>

This reverts commit 2ad5e4a.

Signed-off-by: wep21 <daisuke.nishimatsu1021@gmail.com>

wep21 · 2023-08-23T17:17:06Z

@sunjiahao1999 I'm sorry for late reply. This is the evaluation result with nuscenes dataset after 20 epoch training.

Evaluating bboxes of pred_instances_3d
mAP: 0.5562                                                                                                                                                                                                                                                                  
mATE: 0.3236
mASE: 0.2586
mAOE: 0.3608
mAVE: 0.3053
mAAE: 0.1887
NDS: 0.6344

wep21 · 2023-08-23T17:19:04Z

20230819_092709.log

wep21 · 2023-08-23T17:30:35Z

This figure comes from the original paper of transfusion. The second row shows the score in the same condition as my experiment.

wep21 · 2023-08-25T04:52:11Z

inference demo

python demo/pcd_demo.py demo/data/nuscenes/n015-2018-07-24-11-22-45+0800__LIDAR_TOP__1532402927647951.pcd.bin projects/TransFusion/configs/transfusion_lidar_pillar02_second_secfpn_8xb2-cyclic-20e_nus-3d.py ~/Downloads/mmdet3d_epoch20.pth --show --score-thr 0.05

JingweiZhang12 · 2023-08-28T00:32:45Z

@wep21 Please fix the lint.

Signed-off-by: wep21 <daisuke.nishimatsu1021@gmail.com>

wep21 · 2023-08-28T05:40:41Z

@JingweiZhang12 I fixed lint at 2cb5f78.

wep21 · 2023-09-06T02:40:28Z

@JingweiZhang12 @sunjiahao1999 friendly ping

…latest

friendship1 · 2024-01-18T18:35:41Z

I was looking for this transfusion code. Thank you.

Just to double-check, are the 54.4 and 64.7 values in the table in the paper you posted (#2547 (comment)), 2nd row, left column, the ones you're comparing to (#2547 (comment))? In other words, is it correct that your experiment yielded higher numbers than the paper (just for pillar based, lidar-only)?

Also, I know that the original Transfusion paper used a "fading strategy", and I was wondering if this was applied in your code or in your experiments.

Thanks the great work!

friendship1 · 2024-01-18T23:53:51Z

I just checked that the fading strategy is implemented as a custom hook.

But what you implemented doesn't seem to use a 2-stage learning method: pretrain only the LiDAR model first and then further train this model on additional modality with camera input added.
is that correct?

JingweiZhang12 reviewed May 25, 2023

View reviewed changes

JingweiZhang12 changed the title ~~feat: add transfusion~~ [WIP][Feature]Support TransFusion May 25, 2023

wep21 changed the title ~~[WIP][Feature]Support TransFusion~~ [WIP][Feature] Support TransFusion May 30, 2023

wep21 force-pushed the add-transfusion-latest branch 3 times, most recently from fc37092 to 2ad5e4a Compare June 1, 2023 06:33

wep21 requested a review from JingweiZhang12 June 1, 2023 06:49

JingweiZhang12 reviewed Jun 6, 2023

View reviewed changes

wep21 force-pushed the add-transfusion-latest branch from 004e436 to 18bf4de Compare June 6, 2023 17:48

wep21 force-pushed the add-transfusion-latest branch from fd47bf7 to 4ad9944 Compare July 15, 2023 15:31

wep21 added 12 commits August 18, 2023 15:13

add transfusion as project

ebc1271

Signed-off-by: wep21 <daisuke.nishimatsu1021@gmail.com>

use code from BEVFusion

a44bbfb

Signed-off-by: wep21 <daisuke.nishimatsu1021@gmail.com>

revert changes to original

2e45576

Signed-off-by: wep21 <daisuke.nishimatsu1021@gmail.com>

fix lint

329d077

Signed-off-by: wep21 <daisuke.nishimatsu1021@gmail.com>

override predict method

77509a8

Signed-off-by: wep21 <daisuke.nishimatsu1021@gmail.com>

rename config

2be0004

Signed-off-by: wep21 <daisuke.nishimatsu1021@gmail.com>

rename detector class name

2ff134a

Signed-off-by: wep21 <daisuke.nishimatsu1021@gmail.com>

rename head in bevfusion

51a6945

Signed-off-by: wep21 <daisuke.nishimatsu1021@gmail.com>

Revert "revert changes to original"

c74b50c

This reverts commit 2ad5e4a.

add transfusion head

6b2b6ba

Signed-off-by: wep21 <daisuke.nishimatsu1021@gmail.com>

fix lint

5850065

Signed-off-by: wep21 <daisuke.nishimatsu1021@gmail.com>

update docstring

79415f6

Signed-off-by: wep21 <daisuke.nishimatsu1021@gmail.com>

wep21 force-pushed the add-transfusion-latest branch from 4ad9944 to 79415f6 Compare August 18, 2023 06:13

wep21 requested a review from JingweiZhang12 August 25, 2023 05:11

JingweiZhang12 requested a review from sunjiahao1999 August 28, 2023 00:30

fix lint

2cb5f78

Signed-off-by: wep21 <daisuke.nishimatsu1021@gmail.com>

wep21 force-pushed the add-transfusion-latest branch 2 times, most recently from fe25f7a to 2cb5f78 Compare January 14, 2024 14:44

Merge remote-tracking branch 'upstream/dev-1.x' into add-transfusion-…

2587d4f

…latest



		@TASK_UTILS.register_module()
		class HeuristicAssigner3D(BaseAssigner):

[WIP][Feature] Support TransFusion #2547

Are you sure you want to change the base?

[WIP][Feature] Support TransFusion #2547

Conversation

wep21 commented May 23, 2023

Motivation

Modification

BC-breaking (Optional)

Use cases (Optional)

Checklist

CLAassistant commented May 23, 2023 • edited Loading

JingweiZhang12 commented May 25, 2023

codecov bot commented May 25, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JingweiZhang12 May 31, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JingweiZhang12 Jun 1, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JingweiZhang12 commented May 25, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JingweiZhang12 commented Jun 6, 2023

wep21 commented Jun 6, 2023

JingweiZhang12 commented Jun 7, 2023

sunjiahao1999 commented Aug 16, 2023

wep21 commented Aug 23, 2023 • edited Loading

wep21 commented Aug 23, 2023

wep21 commented Aug 23, 2023

wep21 commented Aug 25, 2023

JingweiZhang12 commented Aug 28, 2023

wep21 commented Aug 28, 2023

wep21 commented Sep 6, 2023

friendship1 commented Jan 18, 2024

friendship1 commented Jan 18, 2024

CLAassistant commented May 23, 2023 •

edited

Loading

codecov bot commented May 25, 2023 •

edited

Loading

JingweiZhang12 May 31, 2023 •

edited

Loading

JingweiZhang12 Jun 1, 2023 •

edited

Loading

JingweiZhang12 commented May 25, 2023 •

edited

Loading

wep21 commented Aug 23, 2023 •

edited

Loading