Low performance on NuScenes Dataset

[magnetmoment]

I have reproduced FSHNet several times, with the batch_size and other configurations completely consistent with the original code. However, the results are unstable and lower than expected. I ran the experiments three times using different seeds, and the results are as follows:
I encountered the same issue when reproducing Scatterformer. Have you experienced this kind of problem before?

2026-01-22 14:27:01,211 INFO ----------------Nuscene detection_cvpr_2019 results-----------------
***car error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.17, 0.15, 0.10, 0.28, 0.19 | 79.60, 88.09, 90.66, 91.70 | mean AP: 0.8751432550916238
***truck error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.32, 0.19, 0.10, 0.27, 0.24 | 42.69, 59.23, 67.25, 72.03 | mean AP: 0.6029768758681523
***construction_vehicle error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.70, 0.44, 0.92, 0.13, 0.28 | 3.43, 17.20, 34.35, 49.82 | mean AP: 0.26200965574198887
***bus error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.29, 0.18, 0.07, 0.45, 0.23 | 57.79, 77.36, 88.43, 91.10 | mean AP: 0.7866938759643214
***trailer error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.52, 0.22, 0.50, 0.21, 0.17 | 13.79, 34.84, 54.86, 64.62 | mean AP: 0.42025192120351107
***barrier error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.19, 0.26, 0.06, nan, nan | 62.78, 72.32, 75.48, 76.62 | mean AP: 0.7180176944759702
***motorcycle error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.19, 0.23, 0.22, 0.45, 0.26 | 66.25, 76.68, 77.65, 78.07 | mean AP: 0.7466188352302552
***bicycle error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.16, 0.25, 0.40, 0.20, 0.01 | 54.56, 57.15, 57.40, 57.79 | mean AP: 0.5672661737185737
***pedestrian error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.13, 0.27, 0.34, 0.22, 0.09 | 86.25, 87.51, 88.56, 89.35 | mean AP: 0.8791682325984863
***traffic_cone error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.12, 0.31, nan, nan, nan | 75.28, 76.17, 77.53, 79.72 | mean AP: 0.7717357754804642
--------------average performance-------------
trans_err: 0.2786
scale_err: 0.2499
orient_err: 0.3019
vel_err: 0.2761
attr_err: 0.1850
mAP: 0.6630
NDS: 0.7023

2026-01-20 18:35:38,778 INFO recall_roi_0.3: 0.000000
2026-01-20 18:35:38,778 INFO recall_rcnn_0.3: 0.821117
2026-01-20 18:35:38,778 INFO recall_roi_0.5: 0.000000
2026-01-20 18:35:38,778 INFO recall_rcnn_0.5: 0.634022
2026-01-20 18:35:38,778 INFO recall_roi_0.7: 0.000000
2026-01-20 18:35:38,778 INFO recall_rcnn_0.7: 0.312125
2026-01-20 18:35:38,781 INFO Average predicted number of objects(6019 samples): 213.981
2026-01-20 18:39:51,768 INFO The predictions of NuScenes have been saved to /root/autodl-tmp/HEDNet-main_trans/tools/output/eval/epoch_36/val/default/final_result/data/results_nusc.json
2026-01-20 18:42:50,110 INFO ----------------Nuscene detection_cvpr_2019 results-----------------
***car error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.16, 0.15, 0.09, 0.27, 0.19 | 80.14, 88.41, 90.95, 92.12 | mean AP: 0.8790625719761649
***truck error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.33, 0.20, 0.08, 0.26, 0.23 | 39.05, 60.29, 68.26, 72.47 | mean AP: 0.600194236052701
***construction_vehicle error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.69, 0.44, 0.83, 0.14, 0.30 | 4.05, 17.45, 35.13, 52.36 | mean AP: 0.27247677572441287
***bus error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.30, 0.18, 0.05, 0.49, 0.22 | 56.80, 78.55, 88.73, 91.01 | mean AP: 0.78774056082737
***trailer error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.52, 0.23, 0.47, 0.22, 0.16 | 12.18, 38.05, 55.67, 67.54 | mean AP: 0.4335911241651135
***barrier error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.18, 0.25, 0.07, nan, nan | 65.51, 74.39, 77.56, 78.74 | mean AP: 0.7405045677135302
***motorcycle error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.18, 0.22, 0.21, 0.48, 0.27 | 66.61, 78.53, 79.60, 80.02 | mean AP: 0.7618981289304854
***bicycle error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.15, 0.25, 0.38, 0.19, 0.02 | 57.98, 60.71, 61.05, 61.47 | mean AP: 0.6030407400024487
***pedestrian error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.12, 0.27, 0.34, 0.21, 0.09 | 87.30, 88.45, 89.40, 90.29 | mean AP: 0.8885940174977791
***traffic_cone error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.11, 0.31, nan, nan, nan | 77.21, 78.24, 79.59, 81.85 | mean AP: 0.7922146065244937
--------------average performance-------------
trans_err: 0.2766
scale_err: 0.2485
orient_err: 0.2802
vel_err: 0.2840
attr_err: 0.1855
mAP: 0.6759
NDS: 0.7105

26-01-21 14:56:37,186 INFO Generate label finished(sec_per_example: 0.0186 second).
2026-01-21 14:56:37,186 INFO recall_roi_0.3: 0.000000
2026-01-21 14:56:37,186 INFO recall_rcnn_0.3: 0.810577
2026-01-21 14:56:37,186 INFO recall_roi_0.5: 0.000000
2026-01-21 14:56:37,186 INFO recall_rcnn_0.5: 0.616258
2026-01-21 14:56:37,186 INFO recall_roi_0.7: 0.000000
2026-01-21 14:56:37,186 INFO recall_rcnn_0.7: 0.299738
2026-01-21 14:56:37,189 INFO Average predicted number of objects(6019 samples): 258.755
2026-01-21 15:01:38,084 INFO The predictions of NuScenes have been saved to /root/autodl-tmp/FSHNet/tools/output/eval/eval_with_train/epoch_36/val/final_result/data/results_nusc.json
2026-01-21 15:05:11,029 INFO ----------------Nuscene detection_cvpr_2019 results-----------------
***car error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.17, 0.15, 0.10, 0.29, 0.20 | 78.83, 87.60, 90.16, 91.33 | mean AP: 0.8698178415137661
***truck error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.34, 0.20, 0.09, 0.26, 0.23 | 39.85, 58.65, 67.50, 71.72 | mean AP: 0.5942878120287329
***construction_vehicle error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.73, 0.45, 0.83, 0.13, 0.33 | 3.57, 15.66, 33.09, 48.71 | mean AP: 0.252573749876297
***bus error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.31, 0.17, 0.08, 0.47, 0.25 | 54.99, 75.80, 87.25, 89.30 | mean AP: 0.7683537925814595
***trailer error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.56, 0.22, 0.49, 0.22, 0.18 | 12.52, 34.76, 52.59, 61.49 | mean AP: 0.4033745579840534
***barrier error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.19, 0.25, 0.07, nan, nan | 66.68, 76.58, 79.96, 81.04 | mean AP: 0.7606421474136371
***motorcycle error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.19, 0.23, 0.23, 0.49, 0.26 | 66.18, 77.54, 78.52, 78.91 | mean AP: 0.7528854429322187
***bicycle error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.16, 0.25, 0.34, 0.21, 0.02 | 55.59, 58.01, 58.25, 58.68 | mean AP: 0.5763427829977583
***pedestrian error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.13, 0.27, 0.36, 0.23, 0.09 | 86.04, 87.28, 88.30, 89.19 | mean AP: 0.8770465046821916
***traffic_cone error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.12, 0.31, nan, nan, nan | 75.48, 76.49, 77.99, 80.30 | mean AP: 0.7756535543336284
--------------average performance-------------
trans_err: 0.2891
scale_err: 0.2501
orient_err: 0.2882
vel_err: 0.2893
attr_err: 0.1936
mAP: 0.6631
NDS: 0.7005

[Say2L]

I provided the checkpoint file for the nuScenes dataset. You can load it in your code to check whether there are any bugs in your code. The detection results have variance, but the loss curve during training epochs is stable. You can compare the loss curve between yours and that in the log file we provided.

[magnetmoment]

I provided the checkpoint file for the nuScenes dataset. You can load it in your code to check whether there are any bugs in your code. The detection results have variance, but the loss curve during training epochs is stable. You can compare the loss curve between yours and that in the log file we provided.

Thank you for your reply. First, we retested using the checkpoint you provided, and the results matched what you described (68.57). The epoch loss was also largely consistent with the log you provided (though it was sometimes slightly lower or higher during the three training runs).
We will continue to investigate the issues during training, but I wanted to ask: Did you encounter unstable results when training FSHNet or Scatterformer or SAFDNet? What kind of fluctuation in mAP did you generally observe?

[Say2L]

I provided the checkpoint file for the nuScenes dataset. You can load it in your code to check whether there are any bugs in your code. The detection results have variance, but the loss curve during training epochs is stable. You can compare the loss curve between yours and that in the log file we provided.

Thank you for your reply. First, we retested using the checkpoint you provided, and the results matched what you described (68.57). The epoch loss was also largely consistent with the log you provided (though it was sometimes slightly lower or higher during the three training runs). We will continue to investigate the issues during training, but I wanted to ask: Did you encounter unstable results when training FSHNet or Scatterformer or SAFDNet? What kind of fluctuation in mAP did you generally observe?

I mainly conducted experiments on the Waymo dataset, where the obtained results have low variance. There is indeed a relatively larger variance in mAP on nuScenes.