Incorporating Historical Frame Points in StreamPETR

[Gigalomanicx]

I am currently working on extending the LiDAR modality within the StreamPETR framework. However, I am unsure how to incorporate historical frame sweep points into the data{} during the forward_train process.

From my understanding, it seems that I need to modify the nuscenes_converter.py file and regenerate the PKL files. Could someone provide guidance on how to achieve this?

[Gigalomanicx]

Also I'm confused about the variable names here, is the gradient being calculated here using the loss of two frames?

StreamPETR/projects/mmdet3d_plugin/models/detectors/petr3d.py

Lines 104 to 137 in 95f6470

	def obtain_history_memory(self,
	gt_bboxes_3d=None,
	gt_labels_3d=None,
	gt_bboxes=None,
	gt_labels=None,
	img_metas=None,
	centers2d=None,
	depths=None,
	gt_bboxes_ignore=None,
	**data):
	losses = dict()
	T = data['img'].size(1)
	num_nograd_frames = T - self.num_frame_head_grads
	num_grad_losses = T - self.num_frame_losses
	for i in range(T):
	requires_grad = False
	return_losses = False
	data_t = dict()
	for key in data:
	data_t[key] = data[key][:, i]
	data_t['img_feats'] = data_t['img_feats']
	if i >= num_nograd_frames:
	requires_grad = True
	if i >= num_grad_losses:
	return_losses = True
	loss = self.forward_pts_train(gt_bboxes_3d[i],
	gt_labels_3d[i], gt_bboxes[i],
	gt_labels[i], img_metas[i], centers2d[i], depths[i], requires_grad=requires_grad,return_losses=return_losses,**data_t)
	if loss is not None:
	for key, value in loss.items():
	losses['frame_'+str(i)+"_"+key] = value
	return losses

StreamPETR/projects/mmdet3d_plugin/models/detectors/petr3d.py

Lines 264 to 271 in 95f6470

	if T-self.num_frame_backbone_grads > 0:
	self.eval()
	with torch.no_grad():
	prev_img_feats = self.extract_feat(prev_img, T-self.num_frame_backbone_grads, True)
	self.train()
	data['img_feats'] = torch.cat([prev_img_feats, rec_img_feats], dim=1)
	else:
	data['img_feats'] = rec_img_feats

#232 (comment)