Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dev/temporal #29

Open
wants to merge 105 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
105 commits
Select commit Hold shift + click to select a range
a402801
Object detection header
sty61010 Nov 24, 2021
0c790d1
oft detector.
sty61010 Dec 30, 2021
87003ca
Correct oft detector.
sty61010 Jan 3, 2022
ebed21f
With seg loss.
sty61010 Jan 4, 2022
603d204
ignore seg info.
sty61010 Jan 4, 2022
7cab9da
Auto name dir.
sty61010 Jan 5, 2022
d3d822b
Fix checkpoints.
sty61010 Jan 13, 2022
acd94cb
Add mmdetection3d/
sty61010 Jan 19, 2022
c361a10
Add gt_bboxes_3d, gt_labels_3d, gt_names_3d.
sty61010 Jan 21, 2022
380569c
mmcollact_fn Done.
sty61010 Jan 23, 2022
28e9e90
Add mmdetection3d
andy94077 Jan 24, 2022
71fa0d9
NMS bug.
sty61010 Jan 25, 2022
5a43775
BEV visualization.
sty61010 Jan 26, 2022
d9ef5b6
Validation visualization.
sty61010 Jan 26, 2022
e4e29f8
Merge visualization
andy94077 Jan 26, 2022
5f2498e
Add backbone
andy94077 Jan 26, 2022
715a029
Test Evaluation.
sty61010 Jan 26, 2022
f835bbd
Update visualization
andy94077 Jan 27, 2022
63723d6
Merge branch 'pp' of https://github.com/sty61010/fiery into pp
andy94077 Jan 27, 2022
1e2aa6e
Control validation samples.
sty61010 Jan 27, 2022
7c3d03a
Add testloader and auto test.
sty61010 Jan 27, 2022
91dd1f3
Update to pytorch-lightning 1.5.9
andy94077 Jan 27, 2022
eb86d5c
Fix argument
andy94077 Jan 27, 2022
1125f2c
Remove ckpt_path argument after trainer.fit
andy94077 Jan 27, 2022
5fca3e1
Alter train.test().
sty61010 Jan 27, 2022
7e82ad0
Fix trainer.test()
sty61010 Jan 27, 2022
33efe1b
Add VAL_BATCHSIZE
andy94077 Jan 28, 2022
1984d77
Enable SEG_USE
andy94077 Feb 7, 2022
5444b08
Update environment.yml
andy94077 Feb 7, 2022
768707e
Merge pull request #1 from sty61010/pp
andy94077 Feb 7, 2022
3981172
Build models from config
andy94077 Feb 7, 2022
f822f28
Remove mmdet3d directory
andy94077 Feb 7, 2022
cc4021b
Enable SEG_USE in lss.yml
andy94077 Feb 7, 2022
7921a82
Update configs
andy94077 Feb 8, 2022
5a70a6c
Merge pull request #2 from sty61010/dev/dynamic-config
andy94077 Feb 8, 2022
00f963a
Add image tag
andy94077 Feb 8, 2022
b052348
Fix seg class chosen labels.
sty61010 Feb 9, 2022
66dd352
Config for twcc.
sty61010 Feb 10, 2022
4bf98d5
Resume ckpt
sty61010 Feb 10, 2022
c5f6928
To do: data augmentation.
sty61010 Feb 11, 2022
0df8f91
Binary Segmentation
sty61010 Feb 11, 2022
a736512
Semantic segmantation.
sty61010 Feb 11, 2022
09fb372
Add centernet
andy94077 Feb 13, 2022
c639a9e
Add centernet and wrappers
andy94077 Feb 14, 2022
f5b5580
Fix seg class chosen labels.
sty61010 Feb 9, 2022
be41d04
Config for twcc.
sty61010 Feb 10, 2022
7529570
Resume ckpt
sty61010 Feb 10, 2022
9c126c2
To do: data augmentation.
sty61010 Feb 11, 2022
567b638
Binary Segmentation
sty61010 Feb 11, 2022
74fb788
Semantic segmantation.
sty61010 Feb 11, 2022
812867a
Merge branch 'master' into dev/centernet
andy94077 Feb 14, 2022
b0bfed5
update resolution.
sty61010 Feb 14, 2022
d271384
seg decoded_bev with backbone.
sty61010 Feb 15, 2022
942d2a4
seg decoded bev parameters in mm backbone.
sty61010 Feb 16, 2022
c1c87b4
Add head mapping
andy94077 Feb 16, 2022
de2edb4
Change centerhead config
andy94077 Feb 16, 2022
9840c1e
Merge branch 'dev/centernet' of https://github.com/sty61010/fiery int…
andy94077 Feb 16, 2022
dcb1647
Modified Efficient backbone layer.
sty61010 Feb 16, 2022
50442fa
Fix config space.
sty61010 Feb 16, 2022
dcf109f
Change voxel size
andy94077 Feb 17, 2022
dda70ad
Fix voxel size
andy94077 Feb 17, 2022
2369928
Merge pull request #5 from sty61010/dev/centernet
andy94077 Feb 17, 2022
eb86066
Merge branch 'master' into dev/image_encoder
andy94077 Feb 17, 2022
1da971f
Merge pull request #6 from sty61010/dev/image_encoder
andy94077 Feb 17, 2022
b7e5023
Set larger dim
andy94077 Feb 17, 2022
ebe0d4e
Change post_center_range
andy94077 Feb 17, 2022
05ea3da
Fix PP parameters.
sty61010 Feb 17, 2022
d909c30
Add trainset
andy94077 Feb 18, 2022
79d691d
Add trainset
andy94077 Feb 18, 2022
5132317
Merge branch 'dev/test-trainset' of https://github.com/sty61010/fiery…
andy94077 Feb 18, 2022
d9e990b
Add visualization during training
andy94077 Feb 18, 2022
cf7726a
modify visualization.
sty61010 Feb 18, 2022
9f5decf
Merge branch 'dev/large-dim' of https://github.com/sty61010/fiery int…
sty61010 Feb 18, 2022
148b29b
merge large-dim
sty61010 Feb 18, 2022
ea61e8a
Merge pull request #8 from sty61010/dev/test-trainset
andy94077 Feb 19, 2022
9753a8f
Merge pull request #7 from sty61010/dev/large-dim
andy94077 Feb 19, 2022
65e65e4
Merge branch 'master' of https://github.com/sty61010/fiery into dev/c…
sty61010 Feb 19, 2022
4401559
modified tb_logging path.
sty61010 Feb 19, 2022
fa8fbd0
Image aug.
sty61010 Feb 21, 2022
5c5656e
Init mm visualize
andy94077 Feb 22, 2022
d3e0ee7
Vis
andy94077 Feb 22, 2022
561b36a
Image aug.
sty61010 Feb 23, 2022
9227866
Visualization mm bbox & nuscene bbox & mm center.
andy94077 Feb 23, 2022
fc6b85a
Delete unused function
andy94077 Feb 24, 2022
c2c2474
Delete unused imports and comments
andy94077 Feb 24, 2022
7d51599
Merge pull request #9 from sty61010/dev/visualization
andy94077 Feb 24, 2022
543b765
Merge branch 'master' into dev/image_aug
sty61010 Feb 25, 2022
07b5db4
Merge pull request #10 from sty61010/dev/image_aug
sty61010 Feb 25, 2022
9f1d180
Clean up oft code
andy94077 Feb 25, 2022
1d3531c
Change out_size_factor
andy94077 Feb 25, 2022
b376a5a
Create new yaml file for out_size_factor
andy94077 Feb 28, 2022
76cfe20
Fix config bug
andy94077 Feb 28, 2022
8202ad2
Remove seg parameters when SEG_USE is False
andy94077 Feb 28, 2022
eb98f59
Reduce internal channels.
sty61010 Mar 2, 2022
8aeb5ae
Test Temporal.
sty61010 Mar 3, 2022
77d49af
Fix temporal frame in Evaluation.
sty61010 Mar 7, 2022
a1e7b98
Fix temporal frame in Evaluation.
sty61010 Mar 7, 2022
a0d4a26
2 frame jump.
sty61010 Mar 16, 2022
f3e9f21
2 frame jump.
sty61010 Mar 16, 2022
947dd01
Eval seg.
sty61010 Mar 28, 2022
1fbeffc
Add velocity suprivised
sty61010 Mar 30, 2022
a3ef59c
Fix Seg evaluation.
sty61010 Mar 31, 2022
53c1a61
Fix iou metrics.
sty61010 Apr 1, 2022
a505886
Without segloss
sty61010 Apr 1, 2022
cc50404
large image.
sty61010 Apr 1, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix temporal frame in Evaluation.
sty61010 committed Mar 7, 2022
commit 77d49af49921288026eb7d5a439199ee0649e6b5
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -7,3 +7,4 @@ output_dir/
*.ipynb_checkpoints
*.ipynb
*.ckpt
*.sh
6 changes: 3 additions & 3 deletions fiery/configs/literature/lss.yml
Original file line number Diff line number Diff line change
@@ -40,9 +40,9 @@ LOSS:

IMAGE:
N_CAMERA: 6
FINAL_DIM: [256, 704]
RESIZE_SCALE: 0.44
TOP_CROP: 140
# FINAL_DIM: [256, 704]
# RESIZE_SCALE: 0.44
# TOP_CROP: 140
IMAGE_AUG: False

LOSS:
8 changes: 4 additions & 4 deletions fiery/configs/literature/lss_mini.yml
Original file line number Diff line number Diff line change
@@ -9,7 +9,7 @@ EVA_DIR: "output_dir/lss_mini"
# EVALUATION: False

GPUS: [0]
EPOCHS: 10
EPOCHS: 2
BATCHSIZE: 2
VAL_BATCHSIZE: 2
N_WORKERS: 4
@@ -40,9 +40,9 @@ LOSS:

IMAGE:
N_CAMERA: 6
FINAL_DIM: [256, 704]
RESIZE_SCALE: 0.44
TOP_CROP: 140
# FINAL_DIM: [256, 704]
# RESIZE_SCALE: 0.44
# TOP_CROP: 140
IMAGE_AUG: False

LOSS:
2 changes: 1 addition & 1 deletion fiery/configs/temporal_single_timeframe.yml
Original file line number Diff line number Diff line change
@@ -5,7 +5,7 @@ TAG: 'temporal_single_timeframe'
BATCHSIZE: 4
# PRECISION: 16

TIME_RECEPTIVE_FIELD: 2
TIME_RECEPTIVE_FIELD: 3
N_FUTURE_FRAMES: 0

MODEL:
74 changes: 57 additions & 17 deletions fiery/data.py
Original file line number Diff line number Diff line change
@@ -104,7 +104,8 @@ def __init__(self, nusc, is_train, cfg):

self.scenes = self.get_scenes()
self.ixes = self.prepro()
self.indices = self.get_indices()
# self.indices = self.get_indices()
self.indices = self.get_mm_indices()

# Image resizing and cropping
self.augmentation_parameters = self.get_resizing_and_cropping_parameters()
@@ -183,6 +184,41 @@ def get_indices(self):

return np.asarray(indices)

def get_mm_indices(self):
indices = []
for index in range(len(self.ixes)):
previous_rec = None
previous_id = None
current_indices = []
for t in range(self.sequence_length):
index_t = index - t
# Going over the dataset size limit.
if index_t >= len(self.ixes) or index_t < 0:
if previous_id is not None:
index_t = previous_id
else:
index_t = index

rec = self.ixes[index_t]

# Check if scene is the same
if (previous_rec is not None) and (rec['scene_token'] != previous_rec['scene_token']):
current_indices.append(previous_id)
else:
current_indices.append(index_t)

previous_rec = rec
previous_id = index_t

# reverse order of current_indices
current_indices.reverse()
print("(current_indices): ", (current_indices))

# if is_valid_data:
indices.append(current_indices)

return np.asarray(indices)

def get_resizing_and_cropping_parameters(self):
original_height, original_width = self.cfg.IMAGE.ORIGINAL_HEIGHT, self.cfg.IMAGE.ORIGINAL_WIDTH
final_height, final_width = self.cfg.IMAGE.FINAL_DIM
@@ -540,13 +576,15 @@ def __getitem__(self, index):
'sample_token',
'z_position', 'attribute',
'gt_bboxes_3d', 'gt_labels_3d', 'gt_names_3d', 'input_metas',
# 'index_t',
]
for key in keys:
data[key] = []

instance_map = {}
# Loop over all the frames in the sequence.
for index_t in self.indices[index]:
# print(index_t)
rec = self.ixes[index_t]
# for i in range(1):
# rec = self.ixes[index]
@@ -555,8 +593,8 @@ def __getitem__(self, index):
segmentation, instance, z_position, instance_map, attribute_label, anns_results = \
self.get_label(rec, instance_map)

# future_egomotion = self.get_future_egomotion(rec, index_t)
future_egomotion = self.get_future_egomotion(rec, index)
future_egomotion = self.get_future_egomotion(rec, index_t)
# future_egomotion = self.get_future_egomotion(rec, index)

data['gt_bboxes_3d'].append(anns_results['gt_bboxes_3d'])
data['gt_labels_3d'].append(anns_results['gt_labels_3d'])
@@ -574,25 +612,27 @@ def __getitem__(self, index):
data['z_position'].append(z_position)
data['attribute'].append(attribute_label)

# data['index_t'].append(index_t)

for key, value in data.items():
if key in ['sample_token', 'centerness', 'offset', 'flow', 'gt_bboxes_3d', 'gt_labels_3d', 'gt_names_3d', 'input_metas', ]:
continue
data[key] = torch.cat(value, dim=0)

# If lyft need to subsample, and update future_egomotions
if self.cfg.MODEL.SUBSAMPLE:
for key, value in data.items():
if key in ['future_egomotion', 'sample_token', 'centerness', 'offset', 'flow']:
continue
data[key] = data[key][::2].clone()
data['sample_token'] = data['sample_token'][::2]

# Update future egomotions
future_egomotions_matrix = pose_vec2mat(data['future_egomotion'])
future_egomotion_accum = torch.zeros_like(future_egomotions_matrix)
future_egomotion_accum[:-1] = future_egomotions_matrix[:-1] @ future_egomotions_matrix[1:]
future_egomotion_accum = mat2pose_vec(future_egomotion_accum)
data['future_egomotion'] = future_egomotion_accum[::2].clone()
# if self.cfg.MODEL.SUBSAMPLE:
# for key, value in data.items():
# if key in ['future_egomotion', 'sample_token', 'centerness', 'offset', 'flow']:
# continue
# data[key] = data[key][::2].clone()
# data['sample_token'] = data['sample_token'][::2]

# # Update future egomotions
# future_egomotions_matrix = pose_vec2mat(data['future_egomotion'])
# future_egomotion_accum = torch.zeros_like(future_egomotions_matrix)
# future_egomotion_accum[:-1] = future_egomotions_matrix[:-1] @ future_egomotions_matrix[1:]
# future_egomotion_accum = mat2pose_vec(future_egomotion_accum)
# data['future_egomotion'] = future_egomotion_accum[::2].clone()

instance_centerness, instance_offset, instance_flow = convert_instance_mask_to_center_and_offset_label(
data['instance'], data['future_egomotion'],
@@ -620,7 +660,7 @@ def to(self, device):


def collate_helper(elems, key):
if key in ['gt_bboxes_3d', 'gt_labels_3d', 'gt_names_3d', 'input_metas', ]:
if key in ['gt_bboxes_3d', 'gt_labels_3d', 'gt_names_3d', 'input_metas', 'sample_token',]:
return elems
else:
return torch.utils.data.dataloader.default_collate(elems)
4 changes: 3 additions & 1 deletion fiery/models/fiery.py
Original file line number Diff line number Diff line change
@@ -176,9 +176,11 @@ def forward(self, image, intrinsics, extrinsics, future_egomotion, future_distri
future_egomotions_spatial[:, :(self.receptive_field - 1)]],
dim=1)
x = torch.cat([x, future_egomotions_spatial], dim=-3)
# print("x.shape: ", x.shape)

#  Temporal model
states = self.temporal_model(x)
# print("states.shape: ", states.shape)

if self.n_future > 0:
# Split into present and future features (for the probabilistic model)
@@ -211,7 +213,7 @@ def forward(self, image, intrinsics, extrinsics, future_egomotion, future_distri
detection_input = future_states.flatten(0, 1)
# cls_scores, bbox_preds, dir_cls_preds = self.detection_head([future_states])
else:
decoder_input = states[:, -1:]
decoder_input = states[:, -1:]
detection_input = states[:, -1:].flatten(0, 1)
# print("detection_input: ", detection_input.shape)
bev_output = {}
8 changes: 4 additions & 4 deletions fiery/models/head_wrappers/Anchor3DHeadWrapper.py
Original file line number Diff line number Diff line change
@@ -80,16 +80,16 @@ def __init__(self,
)

def loss(self, batch, preds_dicts):
gt_bboxes = [item[0] for item in batch['gt_bboxes_3d']]
gt_bboxes = [item[-1] for item in batch['gt_bboxes_3d']]
# print("gt_bboxes: ", gt_bboxes)

gt_labels = [item[0] for item in batch['gt_labels_3d']]
input_metas = [item[0] for item in batch['input_metas']]
gt_labels = [item[-1] for item in batch['gt_labels_3d']]
input_metas = [item[-1] for item in batch['input_metas']]
preds_dicts = preds_dicts + (gt_bboxes, gt_labels, input_metas,)
loss_dict = super().loss(*preds_dicts)
loss_dict = {key: torch.stack(loss_value_list).mean() for key, loss_value_list in loss_dict.items()}
return loss_dict

def get_bboxes(self, batch, preds_dicts):
img_metas = [item[0] for item in batch['input_metas']]
img_metas = [item[-1] for item in batch['input_metas']]
return super().get_bboxes(*preds_dicts, img_metas)
10 changes: 5 additions & 5 deletions fiery/models/head_wrappers/CenterHeadWrapper.py
Original file line number Diff line number Diff line change
@@ -75,19 +75,19 @@ def __init__(self,
self.out_size_factor = train_cfg.out_size_factor

def loss(self, batch, preds_dicts, **kwargs):
gt_bboxes_3d = [item[0] for item in batch['gt_bboxes_3d']]
gt_labels_3d = [item[0] for item in batch['gt_labels_3d']]
gt_bboxes_3d = [item[-1] for item in batch['gt_bboxes_3d']]
gt_labels_3d = [item[-1] for item in batch['gt_labels_3d']]
loss_dict = super().loss(gt_bboxes_3d, gt_labels_3d, preds_dicts, **kwargs)
loss_dict = {key: loss_tensor.mean() for key, loss_tensor in loss_dict.items()}
return loss_dict

def get_bboxes(self, batch, preds_dicts):
img_metas = [item[0] for item in batch['input_metas']]
img_metas = [item[-1] for item in batch['input_metas']]
return super().get_bboxes(preds_dicts, img_metas)

def get_heatmaps(self, batch, preds_dicts):
gt_bboxes_3d = [item[0] for item in batch['gt_bboxes_3d']]
gt_labels_3d = [item[0] for item in batch['gt_labels_3d']]
gt_bboxes_3d = [item[-1] for item in batch['gt_bboxes_3d']]
gt_labels_3d = [item[-1] for item in batch['gt_labels_3d']]
gt_heatmaps, anno_boxes, inds, masks = self.get_targets(gt_bboxes_3d, gt_labels_3d)

preds_heatmaps = dict()
1 change: 1 addition & 0 deletions fiery/models/temporal_model.py
Original file line number Diff line number Diff line change
@@ -49,6 +49,7 @@ def forward(self, x):
x = x.permute(0, 2, 1, 3, 4)
x = self.model(x)
x = x.permute(0, 2, 1, 3, 4).contiguous()
# print("FEATURE\n start from: ", self.receptive_field - 1)
return x[:, (self.receptive_field - 1):]


19 changes: 13 additions & 6 deletions fiery/trainer.py
Original file line number Diff line number Diff line change
@@ -119,13 +119,17 @@ def prepare_future_labels(self, batch):
gt_instance = batch['instance']
future_egomotion = batch['future_egomotion']

# print("LABEL\n segmentation_labels: ", segmentation_labels.shape)
# print("LABEL\n start from: ", self.model.receptive_field - 1)

# Warp labels to present's reference frame
segmentation_labels = cumulative_warp_features_reverse(
segmentation_labels[:, (self.model.receptive_field - 1):].float(),
future_egomotion[:, (self.model.receptive_field - 1):],
mode='nearest', spatial_extent=self.spatial_extent,
).long().contiguous()
labels['segmentation'] = segmentation_labels
# print("LABEL\n segmentation_labels: ", segmentation_labels.shape)
future_distribution_inputs.append(segmentation_labels)

# Warp instance labels to present's reference frame
@@ -316,10 +320,10 @@ def validation_step(self, batch, batch_idx):
self.log(f'val_obj_loss/{key}', value, batch_size=self.cfg.VAL_BATCHSIZE)

output_dict = {'val_loss': loss}

# Visualzation & Evaluation
# tokens = batch['sample_token']
# tokens = [token for tokens_time_dim in tokens for token in tokens_time_dim]
# tokens = [tokens_time_dim[-1] for tokens_time_dim in tokens]
if self.cfg.OBJ.HEAD_NAME == 'mm':
# pred_bboxes_list = self.model.detection_head.get_bboxes(batch, output['detection_output'])
if batch_idx == 0:
@@ -478,10 +482,13 @@ def mm_visualize(self, batch, preds_dicts, pred_bboxes_list=None, tokens=None, g
global_step = self.global_step
if tokens is None:
tokens = batch['sample_token']
tokens = [token for tokens_time_dim in tokens for token in tokens_time_dim]
# print("token: ", len(tokens), len(tokens[0]))
# print("tokens:", tokens)
tokens = [tokens_time_dim[-1] for tokens_time_dim in tokens]
# print("token: ", tokens)

gt_bboxes_3d = [item[0] for item in batch['gt_bboxes_3d']]
gt_labels_3d = [item[0] for item in batch['gt_labels_3d']]
gt_bboxes_3d = [item[-1] for item in batch['gt_bboxes_3d']]
gt_labels_3d = [item[-1] for item in batch['gt_labels_3d']]

# if pred_bboxes_list is not None, use it. Otherwise get bboxes from the detection head
if pred_bboxes_list is None:
@@ -564,7 +571,7 @@ def test_step(self, batch, batch_idx):

# Visualzation & Evaluation
tokens = batch['sample_token']
tokens = [token for tokens_time_dim in tokens for token in tokens_time_dim]
tokens = [tokens_time_dim[-1] for tokens_time_dim in tokens]
if self.cfg.OBJ.HEAD_NAME == 'mm':
pred_bboxes_list = self.model.detection_head.get_bboxes(batch, output['detection_output'])
if batch_idx % 150 == 0:
2 changes: 1 addition & 1 deletion test.sh
Original file line number Diff line number Diff line change
@@ -2,4 +2,4 @@ DIR="$1"
CKPT="$2"
PY_ARGS=${@:3}

python train.py --config "$DIR/hparams.yaml" --eval-path "$DIR/checkpoints/$CKPT" BATCHSIZE 4 VAL_BATCHSIZE 4 DATASET.DATAROOT /home/master/10/cytseng/data/sets/nuscenes N_WORKERS 16 ${PY_ARGS}
python train.py --config "$DIR/hparams.yaml" --eval-path "$DIR/checkpoints/$CKPT" DATASET.DATAROOT /home/master/10/cytseng/data/sets/nuscenes N_WORKERS 10 ${PY_ARGS}