Problems about scenarios convertion

[chrisgao99]

Hello scenarionet team,

Thanks for the great work.

I'm trying to use conver_xx.py under scenario_net directory to prepare some datasets. However, only convert_pg.py works fine and all the other codes will raise errors like lack of modules or files. I wonder if I need to install extra packges before using them. Here are the detailed error message:

(intpenv) bash-4.4$python -m scenarionet.convert_waymo -d waymo1
INFO:scenarionet.converter.waymo.utils:No module named 'tensorflow'
INFO:scenarionet.converter.waymo.utils:
Reading raw data
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/sfs/weka/scratch/baz7dy/INTP/scenarionet/scenarionet/convert_waymo.py", line 68, in <module>
    files = get_waymo_scenarios(waymo_data_directory, args.start_file_index, args.num_files)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sfs/weka/scratch/baz7dy/INTP/scenarionet/scenarionet/converter/waymo/utils.py", line 432, in get_waymo_scenarios
    file_list = os.listdir(waymo_data_directory)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/sfs/weka/scratch/baz7dy/INTP/scenarionet/waymo_origin'

(intpenv) bash-4.4$python -m scenarionet.convert_nuscenes -d nus1
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/sfs/weka/scratch/baz7dy/INTP/scenarionet/scenarionet/convert_nuscenes.py", line 12, in <module>
    from scenarionet.converter.nuscenes.utils import convert_nuscenes_scenario, get_nuscenes_scenarios, \
  File "/sfs/weka/scratch/baz7dy/INTP/scenarionet/scenarionet/converter/nuscenes/utils.py", line 8, in <module>
    from nuscenes.eval.prediction.splits import get_prediction_challenge_split
ModuleNotFoundError: No module named 'nuscenes'

(intpenv) bash-4.4$python -m scenarionet.convert_nuplan -d nup1
WARNING:scenarionet.converter.nuplan.type:Can not import nuplan-devkit: No module named 'nuplan'
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/sfs/weka/scratch/baz7dy/INTP/scenarionet/scenarionet/convert_nuplan.py", line 8, in <module>
    from scenarionet.converter.nuplan.utils import get_nuplan_scenarios, convert_nuplan_scenario
  File "/sfs/weka/scratch/baz7dy/INTP/scenarionet/scenarionet/converter/nuplan/utils.py", line 14, in <module>
    from scenarionet.converter.nuplan.type import get_traffic_obj_type, NuPlanEgoType, set_light_status
ImportError: cannot import name 'NuPlanEgoType' from 'scenarionet.converter.nuplan.type' (/sfs/weka/scratch/baz7dy/INTP/scenarionet/scenarionet/converter/nuplan/type.py)

(intpenv) bash-4.4$python -m scenarionet.convert_argoverse2 -d arg1 --num_scenarios=10
WARNING:scenarionet.converter.argoverse2.type:Can not import av2-devkit: No module named 'av2'
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/sfs/weka/scratch/baz7dy/INTP/scenarionet/scenarionet/convert_argoverse2.py", line 11, in <module>
    from scenarionet.converter.argoverse2.utils import convert_av2_scenario, get_av2_scenarios, preprocess_av2_scenarios
  File "/sfs/weka/scratch/baz7dy/INTP/scenarionet/scenarionet/converter/argoverse2/utils.py", line 17, in <module>
    from av2.datasets.motion_forecasting import scenario_serialization
ModuleNotFoundError: No module named 'av2'

Besides, for the conversion of pg, the original state dimension of metadrive is 19 and after coversion it's 41. May I ask how the state dimension is enlarged or which file should I look into to understand the conversion?

And how can I adjust the parameters of pg maps, like traffic density, map horizon?

[QuanyiLi]

Hi Chris,

Did you try following these installation instructions?

https://scenarionet.readthedocs.io/en/latest/

For the different state dimensions, it is because the ScenarioEnv and MetaDriveEnv use different observation spaces. The former one detects lane lines and road boundaries through pseudo lidar, while the latter one is through geometric calculation. You can find more details here: https://github.com/metadriverse/metadrive/blob/233a3a1698be7038ec3dd050ca10b547b4b3324c/metadrive/obs/state_obs.py#L30

[chrisgao99]

Thank you so much for the guide. For PG, it seems that there's no argument for the metadrive parameters like map, traffic_density, etc. So is there any way to tune these parameters before converting to scenarios?

[QuanyiLi]

You are correct. Currently, we hardcode it at:

scenarionet/scenarionet/converter/pg/utils.py

Line 3 in c6775d7

from metadrive.envs.metadrive_env import MetaDriveEnv

Feel free to modify it to get scenarios you want.

[chrisgao99]

Yes, now I can generate scenarios with settings. But can I generate several datasets of scenarios varying only one parameter? For example, I want to set the traffic_density to be 0.1, 0.2, 0.3 and generate 10 scenarios for each density. If I tune the density and let it generate 3 times, the maps of the 3 datasets will also be different. So is there any way I could set a seed for the repeated generation?

Another question is whether it says that the saved scenarios support imitation learning in the document. However, I haven't figured out how to access the expert rollouts. Are there any instructions on this?

[QuanyiLi]

For the first question, I think the program should work as what you want. That is strange. The data collection program will use this function to collect episodes: https://github.com/metadriverse/metadrive/blob/233a3a1698be7038ec3dd050ca10b547b4b3324c/metadrive/envs/base_env.py#L801

It is clear that each environment will be seeded, so it is not possible to have different map structures given the same seed... Could you try diving into this function and make sure that the same seed results in different map structures across different trials? If it is confirmed, please provide me script to reproduce the issue. and I will look into it.

For the second question, this is a bit confusing. For our internal use, all .pkl files from whatever datasets can be replayed in ScenarioEnv. So we can make our own observation class to determine what kind of information we would like to collect and store the observation trajectory and pose trajectories as imitation learning dataset. In this way, we can do IL with these data.

[chrisgao99]

For the first one, I just use

python -m scenarionet.convert_pg -d pg/0.1
python -m scenarionet.convert_pg -d pg/0.2

to get two datasets with traffic densities equal to 0.1 and 0.2 respectively. Then I just use replay ego car policy to generate corresponding videos.

import os
import torch
import numpy as np
import cv2 
from metadrive.policy.replay_policy import ReplayEgoCarPolicy
from metadrive.component.sensors.rgb_camera import RGBCamera
from metadrive.envs import MetaDriveEnv,ScenarioEnv 
from gymnasium.spaces import Box,Dict


data = "/p/yufeng/INTP/scenarionet/pg/0.2"
env_config2=  {
    # "reactive_traffic": False,
    "use_render": False,
    "agent_policy": ReplayEgoCarPolicy,
    "data_directory": data,
    "num_scenarios": 10,
    "image_observation": True,
    "stack_size": 1,   
    "image_on_cuda": False, 
    "sensors":{
                "rgb_camera": (RGBCamera, 224,224),
            },
    "discrete_action": False,
    "vehicle_config": {
        "image_source": "rgb_camera",
    }
}

class MyWrapper(gym.Wrapper):
    def __init__(self, env):
        super().__init__(env)
        self.observation_space = Dict({
            "image": Box(-0.0, 1.0, (3,224,224), dtype=np.float32),
            "state": self.env.observation_space['state'] #BOX -1,1
        })
        
    def reset(self, **kwargs):
        kwargs.pop('options', None)
        kwargs['seed'] = np.random.randint(0,10)
        obs,info = self.env.reset(**kwargs)

        obs['image'] = obs['image'].squeeze(-1)
        obs['image'] = obs['image'].transpose(2,0,1)
        return obs,info
    
    def step(self, action):
        obs, reward, terminate,truncated, info = self.env.step(action)
        obs['image'] = obs['image'].squeeze(-1).transpose(2,0,1)
        return obs, reward, terminate, truncated, info


def env_creator3(env_config):
    env = MetaDriveEnv({"use_render": False, "image_observation": False,'horizon':10})
    env.reset()
    env.close() 
    print("The bullet env is open and closed")
    env = MyWrapper(ScenarioEnv(env_config))
    return env


if __name__ == "__main__":    
    data = "/p/yufeng/INTP/scenarionet/pg/0.2"
    device = 'cuda' if torch.cuda.is_available() else 'cpu'

    env_config2["data_directory"] = data
    env_config2["agent_policy"] = ReplayEgoCarPolicy
    env_config2["reactive_traffic"] = True

    test_env = env_creator3(env_config2)

    # ckpt_path = "logs/ppo_cnn3/final_model.zip"
    # model = PPO.load(ckpt_path,env=test_env)#.to(device)

    fps = 10
    frame_size = (224,224)
    seed = 0
    while (seed < 10): 
        video_writer = cv2.VideoWriter(f'pg0.2_{seed}.mp4', cv2.VideoWriter_fourcc(*'mp4v'), fps, frame_size)
        try:
            obs,info = test_env.reset(seed=seed)
            for i in range(400):
                action = [0,0]
                obs, r, tm, tc, info = test_env.step(action)
                o = obs['image'].transpose(1,2,0)
                o = o[..., :] * 255
                frame = o.astype('uint8') 
                # print("the frame is: ",frame.shape)
                video_writer.write(frame)
                if info["arrive_dest"]:
                    print("this scenario has {} steps".format(i))
                    break
        finally:
            print("does it arrive dest? ",info["arrive_dest"])
            print(i)
            print(tc)
            print(tm)
            video_writer.release()
            print("The video is saved")
            seed += 1

After comparing the 10 videos from the two datasets, I found out that they have different maps. And I think in convert_pg.py, we don't have parameters to control whether we want same maps or not.

For the second one, I just want to ask how to get the rollouts for imitation learning. For RL, I can use stable baseline like this:

env = ScenarioEnv()
model = PPO(CustomPolicy,env)
model.learn()

,which is straightforward. But if I want to use the "Imitation" repo, I didn't find the rollouts. For example,In the pg data directory

scenarionet/pg/0.2/0.2_0$ ls
 dataset_mapping.pkl                    'sd_pg_MetaDrive v0.4.2.3_PGMap-2.pkl'  'sd_pg_MetaDrive v0.4.2.3_PGMap-6.pkl'
 dataset_summary.pkl                    'sd_pg_MetaDrive v0.4.2.3_PGMap-3.pkl'  'sd_pg_MetaDrive v0.4.2.3_PGMap-7.pkl'
'sd_pg_MetaDrive v0.4.2.3_PGMap-0.pkl'  'sd_pg_MetaDrive v0.4.2.3_PGMap-4.pkl'
'sd_pg_MetaDrive v0.4.2.3_PGMap-1.pkl'  'sd_pg_MetaDrive v0.4.2.3_PGMap-5.pkl'

There are many .pkl files for one scenario. Shall I use pickle load these .pkl files? Or Is there easier way to do imitation learning?

[chrisgao99]

Sorry, I made a mistake. The pg scenarios collected indeed have the same map. They are seeded by the scenario_index in convert_pg_scenario:

https://github.com/metadriverse/scenarionet/blob/c6775d719747df42fc1b5fc2aaf0b4da87b360d5/scenarionet/converter/pg/utils.py#L3

My problem is caused by the wrapper where I changed the env seed in the reset function. Thank you for your patience.

For imitation learning, I know we can use the ReplayEgoCar policy to replay the scenarios and save obs and acts by ourselves. But is there easier method to directly access those data from the saved pkl files? Since it can be replayed, I thought the rollouts should have already been saved.

[QuanyiLi]

The .pkl file is a dictionary, so just load it and you can access the trakcs field which records all trajectories

[chrisgao99]

I tried to open the pkl file but I haven't found the place for obs and actions in a scenario.
For example, I open this pkl file, which should save all the stuff of the pg scenario with seed =0:

sd_pg_MetaDrive v0.4.2.3_PGMap-0.pkl

And I print all the keys in it:

dict_keys(['id', 'version', 'map_features', 'dynamic_map_states', 'length', 'metadata', 'tracks'])

Then all the keys under track field:

dict_keys(['cf4f6e6b-3712-4e6b-92c1-578a5b520a8e', 'f4003097-04b1-4b32-9ca1-69c2f458f469', '8e9ae3c6-311c-4685-82a6-77255f162f75', 'cc5a9bac-5f9e-4ddb-8d69-05df218a78cf', '40074731-8b38-4809-959e-995179ce7222', '784184a6-e12a-4ba5-9cca-3fbb47506de3', '8ed32e63-419c-434b-8904-1a3556d23974', '7080f29d-7d59-471b-9faf-046b53ffe0d0', '53b1d833-d3d6-4982-be8a-b7d052009fbc', '93bf3bcf-cafb-426b-a311-2a526f962d37', '8d935473-d7f0-44d9-a1f1-e4cf4e2d0837', '4ccab3f2-4066-46e6-88d2-75550a1ff578', 'fe2412d6-4503-4202-aa20-df5b5aeafc32', '2122d31f-1440-4fab-9e41-e2fbb6c1d318', 'b2e53224-066b-4369-bc65-5aa7d4c51e48', '06d2c1dc-3545-4eaf-94b4-63d99088cdb3', '28454238-7b6b-46c3-9a2c-8ab0edf44526', 'b787d168-9705-4fea-befc-228f12cfc816', 'becd446e-b92c-4277-b755-ac12a678ef55', 'e2185c0e-1712-427b-b03b-e5162d87de60', '7700d408-af5d-4143-b6d5-4d40a7dc5889', 'b3fd364c-680e-4f2f-b61d-0f7679fc18c6', '14133bea-42f4-444d-a749-ae00ef6425e3', '530aab2b-2b3b-40a4-9597-1ed2af16fecb', 'c1ac9aad-b3f7-4736-a88f-e6a8d1647503', '4f0808d6-8f2a-4bc3-b77a-916f4b5a1cd0', '8321bc13-c05c-4e87-b276-c18dd1229bdb', 'ddcbc115-7d58-4a59-a181-7c24732d63d1', '112f63bd-6343-4652-94da-00feb8ba46b3', '047b4548-f3f6-4fe1-a3a2-084d296fe3c4'])

Now I'm confused, what do these tokens mean? Do they represent different objects in one scenario? I keep exploring the keys under these token field:

dict_keys(['type', 'state', 'metadata'])

And for some tokens, there are action key under state key, but I don't know what do they mean and I didn't see keys like observations.

Could you provide more instructions on how to get expert trajectories with the pkl file?

Or I could also use the ReplayEgoCar policy to replay the scenario and save the obs by myself but how can I get the replay actions in this way?