You will prepare the argoverse dataset, both tracking and forecasting splits included. We recommend you putting the tracking and forecasting into separate directories. For instance, I use directories ~/datasets/argoverse/tracking and ~/datasets/argoverse/forecasting.

Remember to soft-link your data location to ./data/argoverse_forecasting and ./data/argoverse_tracking. The final file structure would be similar to:

./data
    -- argoverse_tracking
        -- train
           xxx   xxx   xxx ...
        -- val
           xxx   xxx   xxx ...
        
    -- argoverse_forecasting
        -- train
            -- data
               xxx.csv, xxx.csv, xxx.csv
        -- val
            -- data
                xxx.csv, xxx.csv, xxx.csv

Argoverse-SF models streaming forecasting by re-purposing the tracking data from Argoverse. Please skip this step if you already have it.

• Download the tracking split from Argoverse Link. You will see 4 *.tar.gz for the training set and 1 *.tar.gz for the validation set.
• Extract the data from compressed files locally. Take tracking_val_v1.1.tar.gz for example:

# Normal extraction
tar -xvf tracking_val_v1.1.tar.gz -C ./

# Exclude the images if you have limited disk space
tar -xvf tracking_val_v1.1.tar.gz --exclude="*.jpg" -C ./

• Move everything out of argoverse_tracking/ and merge the training files

# move everything out
mv argoverse_tracking/* ./

# merge training set
mkdir train
mv train1/* train
mv train2/* train
mv train3/* train
mv train4/* train

In the pretraining step, we will use the forecasting data to train a snapshot-based forecasting model, just like normal forecasters on Argoverse.

• Download the forecasting split from Argoverse Link. You will see 1 *.tar.gz for the training set and 1 *.tar.gz for the validation set.
• Extract the forecasting file locally. Take forecasting_val_v1.1.tar.gz for example, the sript is as below.

tar -xvf forecasting_val_v1.1.tar.gz

We will walk you through:

• Generating the Argoverse-SF benchmark files for evaluation and visualization.
• Generating the information files for dataloading during training and inference, recording the ground truth trajectories and orders of frames.

Please use our ./tools/benchmark/argoverse_sf_creation.py to create the Argoverse-SF benchmark, which supports evaluation. The commands is as below, if you follow our instructions on softlinking the argoverse datasets to ./data as described in [Dataset Preparation]. After this step, you will see eval_cat_val.pkl and eval_cat_train.pkl popping up in ./data/streaming_forecasting.

mkdir ./data/streaming_forecasting

# training set
python tools/benchmark/argoverse_sf_creation.py --data_dir ./data/argoverse_tracking/train --output_dir ./data/streaming_forecasting --save_prefix eval_cat_train --hist_length 20 --fut_length 30 

# validation set
python tools/benchmark/argoverse_sf_creation.py --data_dir ./data/argoverse_tracking/val --output_dir ./data/streaming_forecasting --save_prefix eval_cat_val --hist_length 20 --fut_length 30

If you want any customization, please follow the template below.

# training set
python tools/benchmark/argoverse_sf_creation.py --data_dir $path_to_tracking_train --output_dir $path_to_save_streaming_benchmark --save_prefix eval_cat_train --hist_length $history_length_of_forecasting --fut_length $prediction_horizon 

# validation set
python tools/benchmark/argoverse_sf_creation.py --data_dir $path_to_tracking_val --output_dir $path_to_save_streaming_benchmark --save_prefix eval_cat_val --hist_length $history_length_of_forecasting --fut_length $prediction_horizon 

Please use our ./tools/benchmark/info_file.py to create the information files for Argoverse-SF. We mimic the style in mmdetection and mmdetection3d in organizing the information needed for training, inference, and evaluation.

The command is as below, if you follow our instructions to prepare the paths of data. After running these scripts, you will see infos_train.pkl and infos_val.pkl in ./data/streaming_forecasting.

# training set
python tools/benchmark/info_file.py --save_prefix infos_train --benchmark_file eval_cat_train.pkl --data_dir ./data/argoverse_tracking/train

# validation set
python tools/benchmark/info_file.py --save_prefix infos_val --benchmark_file eval_cat_val.pkl --data_dir ./data/argoverse_tracking/val

If you want any customization, please follow the template below.

# training set
python tools/benchmark/info_file.py --data_dir $path_to_tracking_train --output_dir $path_to_save_streaming_benchmark --save_prefix infos_train --benchmark_file eval_cat_train.pkl --hist_length $history_length_of_forecasting --fut_length $prediction_horizon  

# validation set
python tools/benchmark/info_file.py --data_dir $path_to_tracking_validation --output_dir $path_to_save_streaming_benchmark --save_prefix infos_val --benchmark_file eval_cat_val.pkl --hist_length $history_length_of_forecasting --fut_length $prediction_horizon  

To evaluate on Argoverse-SF, the result file is .pkl file compressing a python list, where each item in the list is the result of one frame. For reference, you can check our VectorNet result file for a rough sense.

Specifically, you would use the information file generated before infos_val.pkl. With each item in infos_val.pkl, your result file should also be a list corresponding to each sample in infos_val.pkl. Please note that the trajectories are in the world coordinate.

result = 
[
  # Result for sample[0] in infos_val.pkl 
  {
    ...
  },
  ...
  # Result for sample[i] in infos_val.pkl
  {
    # sequence name in Argoverse, 
    # you should directly copy from sample[i]['seq_name']
    'seq_name': str, 
    # city_name in Argoverse,
    # you should directly copy from sample[i]['city_name']
    'city_name': str,
    # frame number in the sequence,
    # you should directly copy from sample[i]['frame_index']
    'frame_index': int,
    # results of your predictions
    # dictonary with keys being every agent id specified in sample[i]['query_keys']
    'results': {
      # key0 in sample[i]['query_keys]
      sample[i]['query_keys'][0]: ...,
      ...
      # keyj in sample[i]['query_keys]
      sample[i]['query_keys'][j]: {'trajs': numpy array shaped [6, 30, 2], 'confidences': numpy array shaped [6]},
      ...
      sample[i]['query_keys'][-1]: ...
    }
  },
  ...
  # Result for sample[-1] in infos_val.pkl
  {
    ...
  },
]

To evaluate the inference results, run the following command and you will find the metrics.json in you specified $directory_to_save_metric_file

python tools/evaluation.py --result_file $path_streaming_inference.pkl --metric_file_prefix $directory_to_save_metric_file

For example, I want to evaluate my VectorNet results and save to ./results/VectorNet, then:

python tools/evaluation.py --result_file ./results/VectorNet/streaming_inference.pkl --metric_file_prefix ./results/VectorNet/

We structure the metric results in the following way:

• The items of ade6, fde6, and mr6 means the minADE, minFDE and miss rate for K=6, where the forecaster predicts 6 trajectories for each agent.
• We divide 4 groups of agents to acquire the above metrics. The groups are organized according to moving (move) and static (stay) motion, and visible (vis) and occluded (occ) states.
• Finally, the items begin with "ave-" are the major metrics that come from averaging all the groups.

We use tools/inference.py to run streaming forecasting iteratively on every frame. Its template is as below. After running the command, a streaming_inference.pkl will appear at the directory of $path_to_save_results, which is the result file.

python tools/inference.py --config $config_path --weight_path $your_checkpoint --save_prefix $path_to_save_results

For example, if you use our provided VectorNet checkpoint and save the results to ./results/VectorNet, you could simply run as below.

python tools/inference.py --config configs/streamer/config.yaml --weight_path ./ckpts/vectornet.ckpt --save_prefix ./results/VectorNet/

If you have followed the previous steps, especially the paths to data. Training and evaluating VectorNet on Argoverse's forecasting training/validation sets are as simple as:

# training
python tools/pretrain/train_forecaster_vectornet.py --exp_name $your_experiment_name --model_save_dir $directory_to_save

# validation
python tools/pretrain/eval_forecaster_vectornet.py --weight_path $path_to_trained_model

For example, my command is as simple as:

# training
# use wandb for logging
python tools/pretrain/train_forecaster_vectornet.py --exp_name pretrain --model_save_dir ./results --wandb

# validation
python tools/pretrain/eval_forecaster_vectornet.py --weight_path vectornet.ckpt

The expected results of the validation process is similar to below. We focus on minADE, minFDE, and MR.

------------------------------------------------
Prediction Horizon : 30, Max #guesses (K): 6
------------------------------------------------
{'minADE': 0.7742552296892377, 'minFDE': 1.1925503502421884, 'MR': 0.12593737332792865, 'p-minADE': nan, 'p-minFDE': nan, 'p-MR': 0.8569636313846993, 'brier-minADE': 6.351643563315018, 'brier-minFDE': 6.769938683867964, 'DAC': 0.9879999324415611}
------------------------------------------------
------------------------------------------------
Prediction Horizon : 30, Max #guesses (K): 1
------------------------------------------------
{'minADE': 1.53590666902302, 'minFDE': 3.373533234667678, 'MR': 0.5553810295905959, 'p-minADE': 1.53590666902302, 'p-minFDE': 3.373533234667678, 'p-MR': 0.5553810295905959, 'brier-minADE': 1.53590666902302, 'brier-minFDE': 3.373533234667678, 'DAC': 0.9887515200648561}
------------------------------------------------

Suppose you need to customize the training process, such as path to data or optimization details, change the configuration files:

# batch size
batch_size: 32 
val_batch_size: 32 

# number of workers in the dataloader
workers: 4
val_workers: 4

# path to the forecasting data
train_dir: ./data/argoverse_forecasting/train/data/
val_dir: ./data/argoverse_forecasting/val/data/
train_map_dir: null
val_map_dir: null

# use all the training data
ratio: 1.0

# perception range [xxyy]
pred_range:
  - -100.0
  - 100.0
  - -100.0
  - 100.0

# beginning epoch
epoch: 0
optimizer: adam
# total epoch
num_epochs: 24
# frequency of saving checkpoints
save_freq: 1.0

# base learning rate
lr: 0.0005
weight_decay: 1.0e-4
# iterations for warmup
warmup_iters: 1000
warmup_ratio: 1.0e-3
grad_clip: 1.0e-1

# when to drop the learning rate
lr_decay_epoch:
  - 16
  - 20
# ratio of dropping learning rate
lr_decay_rate: 0.25

Suppose you have a pretrained forecaster, immitate how our streaming_vectornet.py turn a regular VectorNet in VectorNet.py into a streaming version of VectorNet. The key part is to use instances for storing the temporal information. Then we can finetune the new streaming model by:

# use --wandb if you have a wandb logger to use
python tools/train_streaming.py --config $config_path --weight_path $pretrained_forecaster --name $exp_name --save_prefix $directory_to_save_results

Specifically, if you follow the previous paths, you can finetune from the pretrained VectorNet provided by us with the following link:

python tools/train_streaming.py --config ./configs/streamer/config.yaml --weight_path ./ckpts/vectornet.ckpt --name finetune --save_prefix ./results/

After this, you can see several checkpoints popping up in ./results/finetune/ckpts/. The final one is the finetuned model.