Question about Closed Simulation.

[Yyb-XJTU]

When I try to Simulation with the below script：

python run_nuplan_simulation.py \
--test_type closed_loop_nonreactive_agents \
--data_path {SIMULATION_DATA} \
--map_path {MAP_PATH} \
--model_path {MODEL_CHECKPOINT_PATH} \
--split_filter_yaml nuplan_simulation/test14_hard.yaml \
--max_scenario_num 10000 \
--batch_size 8 \
--device cuda \
--exp_folder TestHard14_MixtralM_CKS_SepLoss_AugCur50Pct_PastProj_S6_bf16_Jun28_ckpt150k\
--processes-repetition 8

I found that "run_nuplan_simulation.py" does not support the above argument, but "run_simulation.py" does. Is this a mistake in the README description?

[Yyb-XJTU]

After the simulation, I got a very low score:

Additional instructions:
checkpoint: checkpoint-66000 (from your huggingface)
split_filter_yaml：nuplan_simulation/test14_hard.yaml

[AQingRoad]

@Yyb-XJTU 您好！你所### 使用的数据集是直接从Hugging Face Dataset上面下载的吗？能分享一下data_path,map_path的路径吗？我运行这个命令

python run_simulation_closed.py \
--test_type closed_loop_nonreactive_agents \
--data_path /home/linux/Desktop/nuplan-v1.1_STR_2/index/val/val14x15_index/ \
--map_path /home/linux/Desktop/nuplan-v1.1_STR_2/map/ \
--model_path /home/linux/Desktop/str_open/cpks100m/checkpoint-66000/ \
--split_filter_yaml nuplan_simulation/test14_hard.yaml \
--max_scenario_num 10000 \
--batch_size 8 \
--device cuda \
--exp_folder TestHard14_MixtralM_CKS_SepLoss_AugCur50Pct_PastProj_S6_bf16_Jun28_ckpt150k

有下面的问题

Folder where all results are stored: testing_log/closed_loop_nonreactive_agents/str_closed_planner/TestHard14_MixtralM_CKS_SepLoss_AugCur50Pct_PastProj_S6_bf16_Jun28_ckpt150k

Extracting scenarios... False
Filtering with yaml file...
No log files found! This may mean that you need to set your environment, or that all of your log files got filtered out on this worker.
Got scenarios  0
Running simulations...
total scenarios: 0
locally deal with split: [0, 0]
local scenarios: 0
Traceback (most recent call last):
  File "/home/linux/Desktop/StateTransformer/run_simulation_closed.py", line 905, in <module>
    main(args)
  File "/home/linux/Desktop/StateTransformer/run_simulation_closed.py", line 842, in main
    build_simulation_in_batch_multiprocess(experiment_name, scenarios, output_dir, simulation_dir, metric_dir, args.batch_size, args=args)
  File "/home/linux/Desktop/StateTransformer/run_simulation_closed.py", line 528, in build_simulation_in_batch_multiprocess
    assert len(scenario_groups) > 0, f'no scenarios after filtering, check folder and yaml file.'
AssertionError: no scenarios after filtering, check folder and yaml file.

非常感谢！

[JohnZhan2023]

@Yyb-XJTU Thank you for pointing out the mistakes. Actually, the run_simulation.py doesn't contain the post process programs. The run_simulation_closed.py is the correct one. We will update the README.md.

[Yyb-XJTU]

@JohnZhan2023 Thank you for your reply. When I run "run_simulation_closed.py", I find that the result is very close to the result in your paper. However, this file runs "str_closed_planner" (Is this the Planner used in the paper?), and the main body of this planner is "PDM Planner". How does this reflect the role of "str"?

[JohnZhan2023]

@Yyb-XJTU Actually, the str_closed_planner is the postprocessed version of the str planner and it will use pdm's strategy to refine the trajectory given by str. And the pure str (run_simulation.py) will get a higher score when the number of parameters come to 800m as more parameters bring more generalization ability. 800m str will be much better than the 100m. For more details about the str's role, you can resort to the paper.

[Yyb-XJTU]

Thank you for your reply. I have carefully read your paper and noticed that there doesn’t seem to be a detailed explanation of the post-processing steps. Also, in the comparative experiments, it’s not entirely clear which results correspond to pure STR outputs and which are post-processed. If possible, could you please share more details about these experimental results?

In addition, I ran the pure str (100m) alone, and the closed-loop score was only over 30. Is this too low?

[JohnZhan2023]

Actually, in the paper, we don't show the scores of pure STR outputs. All the results are based on the post processing of the PDM. Current simulation pipelines may not support pure STR simulation. The 30+ score is not reasonable apparently and there must be mismatch in the simulation pipeline. You may need to revise the planner and if you are interested in the score of pure STR, you can revise the planner and push it to a new branch, and we can help you debug it.

[JohnZhan2023]

Here is our description of the post processing in the paper.

[larksq]

Thank you for your reply. I have carefully read your paper and noticed that there doesn’t seem to be a detailed explanation of the post-processing steps. Also, in the comparative experiments, it’s not entirely clear which results correspond to pure STR outputs and which are post-processed. If possible, could you please share more details about these experimental results?

In addition, I ran the pure str (100m) alone, and the closed-loop score was only over 30. Is this too low?

We appreciate your interest in our work. The current code base (the planner and model part) was not tested and debuged for "pure STR outputs" against the closed-loop simulation testings without "post-processing steps". So in short, no results can be confirmed or verified, but 30 is way too low to make sense. (even lower than early UrbanDrivers).

It is not clear how you tested. As @JohnZhan2023 mentioned, there might be some mismatches here and there (such as a frequency downsampling on the labels we did for efficiency). We kindly suggest:

1. Empirical good results might need you to retrain from scratch, starting from generating labels without any downsampling with the original NuPlan features used in previous works to make sure everything matches (training with simulations).
2. Or another less reliable walkaround is to change the code of the planner carefully. I would recommend you keep the PDM structure and replace the sampling part from randomly sampling nearby proposals with N identical proposals the same as the model's output. But, again, no results can be guaranteed.

Sharing your codes and detailed results will be helpful and is always welcome.