Can't read Waymo Log Files

[MarkusKramer1]

Hi,
thanks for the great work.
I would like to explore your simulation using the Waymo Open Dataset. Therefore I downloaded the "testing_interactive" part of the dataset. I updated the path in the config.py before starting "python sim.py --max_scenarios 5".

However I get a parsing error for the dataset files. Here is the full output log:

2023-12-05 18:01:04.319057: I tensorflow/core/util/port.cc:111] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-12-05 18:01:04.346737: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2023-12-05 18:01:04.346766: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2023-12-05 18:01:04.346773: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2023-12-05 18:01:04.350884: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: SSE4.1 SSE4.2 AVX AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
Data Loader Initialized Waymo:  0 0 0 None 0 None 150
2023-12-05 18:01:06,INFO:The hyperparameter of gym environment is list as blow
2023-12-05 18:01:06,INFO:frame rate is :10,
DriveEnv Environment initialized
Loading file from:  /media/kramer/Extreme SSD/waymo_dataset/testing_interactive/testing_interactive_tfexample.tfrecord-00000-of-00150  with index of  0
2023-12-05 18:01:06.722136: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-12-05 18:01:06.852966: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-12-05 18:01:06.853097: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-12-05 18:01:06.853932: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-12-05 18:01:06.854025: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-12-05 18:01:06.854087: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-12-05 18:01:07.222556: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-12-05 18:01:07.222683: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-12-05 18:01:07.222763: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-12-05 18:01:07.222821: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1886] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 9901 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 3060, pci bus id: 0000:01:00.0, compute capability: 8.6
 with  279  scenarios and current is  0
2023-12-05 18:01:09.737639: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: roadgraph_samples/xyz.  Can't parse serialized Example.
Traceback (most recent call last):
  File "/home/kramer/honda/InterSim/simulator/sim.py", line 243, in <module>
    main(args_p)
  File "/home/kramer/honda/InterSim/simulator/sim.py", line 97, in main
    ended = env.reset(output_dir=output_dir)
  File "/home/kramer/miniforge3/envs/inter_sim/lib/python3.9/site-packages/gym-0.18.3-py3.9.egg/gym/wrappers/time_limit.py", line 25, in reset
    return self.env.reset(**kwargs)
  File "/home/kramer/honda/InterSim/simulator/interactive_sim/envs/drive.py", line 489, in reset
    loaded_scenario, new_files_loaded = self.data_loader.get_next(filter_config=self.filter_config,
  File "/home/kramer/honda/InterSim/simulator/dataloader/DataLoaderWaymo.py", line 1111, in get_next
    data_to_return = self.get_datadic_fromTFRecord(data, process_intersection=process_intersection,
  File "/home/kramer/honda/InterSim/simulator/dataloader/DataLoaderWaymo.py", line 626, in get_datadic_fromTFRecord
    parsed = tf.io.parse_single_example(data, features_description)
  File "/home/kramer/miniforge3/envs/inter_sim/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/home/kramer/miniforge3/envs/inter_sim/lib/python3.9/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped__ParseExampleV2_Tdense_57_num_sparse_0_ragged_split_types_0_ragged_value_types_0_sparse_types_0_device_/job:localhost/replica:0/task:0/device:CPU:0}} Key: roadgraph_samples/xyz.  Can't parse serialized Example. [Op:ParseExampleV2]

Do you know what the issue could be?

[AnthonyWithLi]

Hi, I have just encountered the same problem. Have you find the solution? I'll be more than grateful if you can share with me!

[larksq]

I guess you are using the tf.example files. There are two types of data from the Waymo Open Dataset. You guys should load the scenario files but not the tf.example files for our dataloader. Let me know if that's the problem.

[AnthonyWithLi]

Yes, I followed your advice to solve the above problem, but then encountered the following error when running sim.py. Do you know how to resolve it? Thank you!

2024-05-08 23:28:48.312803: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at example_parsing_ops.cc:94 : INVALID_ARGUMENT: Feature: scenario/id (data type: string) is required but could not be found.
Traceback (most recent call last):
  File "/home/anthony/Desktop/InterSim-main/simulator/sim.py", line 243, in <module>
    main(args_p)
  File "/home/anthony/Desktop/InterSim-main/simulator/sim.py", line 50, in main
    ended = env.reset(output_dir=output_dir)
  File "/home/anthony/anaconda3/envs/myenv/lib/python3.9/site-packages/gym-0.18.3-py3.9.egg/gym/wrappers/time_limit.py", line 25, in reset
    return self.env.reset(**kwargs)
  File "/home/anthony/Desktop/InterSim-main/simulator/interactive_sim/envs/drive.py", line 489, in reset
    loaded_scenario, new_files_loaded = self.data_loader.get_next(filter_config=self.filter_config,
  File "/home/anthony/Desktop/InterSim-main/simulator/dataloader/DataLoaderWaymo.py", line 1100, in get_next
    parsed = tf.io.parse_single_example(data, features_description_scenario)
  File "/home/anthony/anaconda3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/home/anthony/anaconda3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/execute.py", line 52, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped__ParseExampleV2_Tdense_2_num_sparse_0_ragged_split_types_0_ragged_value_types_0_sparse_types_0_device_/job:localhost/replica:0/task:0/device:CPU:0}} Feature: scenario/id (data type: string) is required but could not be found. [Op:ParseExampleV2]