Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem with training on outdoor dataset nuScenes. #57

Open
fang196 opened this issue Jan 3, 2024 · 2 comments
Open

Problem with training on outdoor dataset nuScenes. #57

fang196 opened this issue Jan 3, 2024 · 2 comments

Comments

@fang196
Copy link

fang196 commented Jan 3, 2024

I'm testing 3DETR in outdoor senerio (nuScenes 3D dataset), but met the same promblem as #28 . The metrics are all zero after training 90 epochs. I visualize the gt and pred boxes as follows. Also, I've found loss_cardinality is unsually high, showing that almost all queries are predicted as objects.

Ground truth:
231226_vis_gt_boxes
Predictions:
231226_vis_pred_boxes

Following are logs:
Epoch [0/90]; Iter [0/3150]; Loss 57.61; LR 1.00e-06; Iter time 11.74; ETA 10:16:18; Mem 25411.98MB
Epoch [0/90]; Iter [10/3150]; Loss 58.18; LR 1.68e-05; Iter time 4.36; ETA 3:48:15; Mem 25499.76MB
Epoch [0/90]; Iter [20/3150]; Loss 57.50; LR 3.27e-05; Iter time 4.03; ETA 3:30:28; Mem 25499.76MB
Epoch [0/90]; Iter [30/3150]; Loss 55.73; LR 4.85e-05; Iter time 3.45; ETA 2:59:22; Mem 25499.76MB
.......
Epoch [89/90]; Iter [3120/3150]; Loss 17.18; LR 1.11e-06; Iter time 3.35; ETA 0:01:40; Mem 25499.76MB
Epoch [89/90]; Iter [3130/3150]; Loss 17.07; LR 1.05e-06; Iter time 2.49; ETA 0:00:49; Mem 25499.76MB
Epoch [89/90]; Iter [3140/3150]; Loss 17.37; LR 1.01e-06; Iter time 2.45; ETA 0:00:24; Mem 25499.76MB
Training Finished.
====================Final Eval Numbers.
mAP0.25, mAP0.50: 0.00, 0.00
AR0.25, AR0.50: 0.00, 0.00

IOU Thresh=0.25
car Average Precision: 0.00
truck Average Precision: 0.00
trailer Average Precision: 0.00
bus Average Precision: 0.00
construction_vehicle Average Precision: 0.00
bicycle Average Precision: 0.00
motorcycle Average Precision: 0.00
pedestrian Average Precision: 0.00
traffic_cone Average Precision: 0.00
barrier Average Precision: 0.00
other Average Precision: 0.00
car Recall: 0.00
truck Recall: 0.00
trailer Recall: 0.00
bus Recall: 0.00
construction_vehicle Recall: 0.00
bicycle Recall: 0.00
motorcycle Recall: 0.00
pedestrian Recall: 0.00
traffic_cone Recall: 0.00
barrier Recall: 0.00
other Recall: 0.00

@fang196 fang196 changed the title Problem with training in outdoor dataset nuScenes. Problem with training on outdoor dataset nuScenes. Jan 4, 2024
@Liuliuliuyikou
Copy link

Hello, how did you solve it in the end? I encountered the same problem.

@Serzhanov
Copy link

@fang196 Did you solve it? I got the same problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants