Training dataset preprocessing

[saeidnp]

Hi,

I'm trying to run the training code, and the README directs users to the Diffusion Planner repository for dataset preprocessing:

Convert nuplan data into npz and generate corresponding data list json file as indicated in https://github.com/ZhengYinan-AIR/Diffusion-Planner.

However, it seems there is an incompatibility between the datasets generated by the Diffusion Planner preprocessing code and what Flow Planner expects. For example, the ego_current_state array in the Diffusion Planner–generated .npz files is 10-dimensional, while the Flow Planner model expects a 16-dimensional array. This mismatch is also reflected in the differing mean/std vector shapes shown in the Diffusion Planner normalization file (linked here) and in the Flow Planner normalization stats (linked here).

I also noticed that the data_process directory in this repo is very similar (but not identical) to the one in Diffusion Planner. Am I correct in assuming that dataset preprocessing for Flow Planner requires scripts analogous to data_process.py and data_process.sh from the Diffusion Planner repository? If so, could you include the appropriate preprocessing scripts for this repo as well?

[kumarmanas]

I had the similar observation. In my observation if we use the same normalization file as in "Diffusion Planner normalization file", then nuplan val 14 results are very bad, also train loss is much higher. Now I am running the training with new file and loss seems to be much better. Once I test the val 14 with new model weights, I will update my results for you.

[ZhuPiter-1mol]

I had the similar observation. In my observation if we use the same normalization file as in "Diffusion Planner normalization file", then nuplan val 14 results are very bad, also train loss is much higher. Now I am running the training with new file and loss seems to be much better. Once I test the val 14 with new model weights, I will update my results for you.

Hello,I meet the same problem, have you resolved it?

[0ttwhy4]

Hi,

I'm trying to run the training code, and the README directs users to the Diffusion Planner repository for dataset preprocessing:

Convert nuplan data into npz and generate corresponding data list json file as indicated in https://github.com/ZhengYinan-AIR/Diffusion-Planner.

However, it seems there is an incompatibility between the datasets generated by the Diffusion Planner preprocessing code and what Flow Planner expects. For example, the ego_current_state array in the Diffusion Planner–generated .npz files is 10-dimensional, while the Flow Planner model expects a 16-dimensional array. This mismatch is also reflected in the differing mean/std vector shapes shown in the Diffusion Planner normalization file (linked here ) and in the Flow Planner normalization stats (linked here ).

I also noticed that the data_process directory in this repo is very similar (but not identical) to the one in Diffusion Planner. Am I correct in assuming that dataset preprocessing for Flow Planner requires scripts analogous to data_process.py and data_process.sh from the Diffusion Planner repository? If so, could you include the appropriate preprocessing scripts for this repo as well?

Thanks for pointing out. The similar issues has been proposed in #3, and we have updated the model preprocessing script accordingly to adapt to the data format. Please checkout the latest update! :)

[saeidnp]

I believe #3 is a different issue. I have already checked out to the latest commit and the issue I reported here still exists. The problem here is that the data pre-processed by the scripts from the diffusion planner repo has a different shape than the model expects in this repo.