Driving Smarts 2023 environments

.gitignore

@@ -120,12 +120,17 @@ build.db
 
 # run logs
 **/examples/**/logs
+**/examples/**/videos
 _run_logs
 results
 supervisord.log
 supervisord.pid
 xdummy.log
 
+# Argoverse dataset
+log_map_archive_[a-z0-9-]*.json
+scenario_[a-z0-9-]*.parquet
+
 # replay data
 *.jsonl
 

CHANGELOG.md

@@ -13,9 +13,11 @@ Copy and pasting the git commit messages is __NOT__ enough.
 - Added an actor capture manager interface, `ActorCaptureManager`, which describes a manager that handles the change of control of actors. Operations in an actor manager step should not cause conflict in the simulation.
 - Added a new entry tactic, `IdEntryTactic`, which provides the scenario the ability to select a specific actor for an agent to take over.
 - Registered a new `chase-via-points-agent-v0` agent in agent zoo, which can effectively chase via points across different road sections by using the waypoints.
+- Added new driving-smarts-v2023 benchmark consisting of new (i) driving-smarts-v2023 env and (ii) platoon-v0 env.
 ### Changed
 - The trap manager, `TrapManager`, is now a subclass of `ActorCaptureManager`.
 - Considering lane-change time ranges between 3s and 6s, assuming a speed of 13.89m/s, the via sensor lane acquisition range was increased from 40m to 80m, for better driving ability.
+- Modified naming of benchmark used in NeurIPS 2022 from driving-smarts-competition-env to driving-smarts-v2022.
 ### Deprecated
 ### Fixed
 - Fixed an issue where Argoverse scenarios with a `Mission` would not run properly.

docs/benchmarks/agent_zoo.rst

@@ -50,19 +50,19 @@ Available zoo agents
      - Remarks
    * - | zoo.policies:interaction-aware-motion-prediction-agent-v0
        | zoo/policies/interaction_aware_motion_prediction
-     - driving_smarts==0.0
+     - driving_smarts_2022==0.0
      - :attr:`~smarts.core.controllers.ActionSpaceType.TargetPose`
      - `code <https://github.com/smarts-project/smarts-project.rl/tree/master/interaction_aware_motion_prediction>`__
      - Contributed as part of `NeurIPS 2022 Driving SMARTS <https://smarts-project.github.io/archive/2022_nips_driving_smarts/>`__ competition.
    * - | zoo.policies:control-and-supervised-learning-agent-v0
        | zoo/policies/control_and_supervised_learning
-     - driving_smarts==0.0
+     - driving_smarts_2022==0.0
      - :attr:`~smarts.core.controllers.ActionSpaceType.TargetPose`
      - `code <https://github.com/smarts-project/smarts-project.rl/tree/master/control_and_supervised_learning>`__
      - Contributed as part of `NeurIPS 2022 Driving SMARTS <https://smarts-project.github.io/archive/2022_nips_driving_smarts/>`__ competition.
    * - | zoo.policies:discrete-soft-actor-critic-agent-v0
        | zoo/policies/discrete_soft_actor_critic
-     - driving_smarts==0.0
+     - driving_smarts_2022==0.0
      - :attr:`~smarts.core.controllers.ActionSpaceType.TargetPose`
      - `code <https://github.com/smarts-project/smarts-project.rl/tree/master/discrete_soft_actor_critic>`__
      - Contributed as part of `NeurIPS 2022 Driving SMARTS <https://smarts-project.github.io/archive/2022_nips_driving_smarts/>`__ competition.

docs/benchmarks/benchmark.rst

@@ -24,28 +24,32 @@ benchmark version, then the benchmark's latest version is run by default.
 
 .. code:: bash
 
-   $ scl benchmark run driving_smarts smarts.zoo:random-relative-target-pose-agent-v0 --auto-install 
-   Starting `Driving SMARTS V1` benchmark.
+   $ scl benchmark run driving_smarts_2022 smarts.zoo:random-relative-target-pose-agent-v0 --auto-install 
+
+   <-- Starting `Driving SMARTS 2022` benchmark -->
+
    This is a cleaned up version of the Driving SMARTS benchmark.
 
        Using `TargetPose` agent action has an applied 28m/s cap for agent motion.
        Using `RelativeTargetPose` agent action, the constraint is inbuilt into the action space.
 
        For history see: 
            - https://codalab.lisn.upsaclay.fr/competitions/6618
-           - https://smarts-project.github.io/archive/2022_nips_driving_smarts/
+           - https://smarts-project.github.io/archive/2022_nips_driving_smarts/competition/
+
    Evaluating 1_to_2lane_left_turn_c...
    Evaluating 3lane_merge_multi_agent...
    ...
    Scoring 1_to_2lane_left_turn_c...
-   Evaluation complete...
 
-   `Driving SMARTS V0` result:
-   - completion: 1
-   - humanness: 0.7
-   - rules: 0.9
-   - time: 0.2
-   - overall: 0.504
+   SCORE
+   {'overall': 0.424,
+    'dist_to_destination': 0.925,
+    'humanness': 0.769,
+    'rules': 1.0,
+    'time': 0.265}
+
+   <-- Evaluation complete -->
 
 See available benchmarks
 ------------------------
@@ -56,7 +60,7 @@ The ``scl benchmark list`` command can be used to see the list of available benc
 
    $ scl benchmark list 
    BENCHMARK_NAME               BENCHMARK_ID             VERSIONS
-   - Driving SMARTS:            driving_smarts           0.0 0.1
+   - Driving SMARTS 2022:       driving_smarts_2022      0.0 0.1
 
 Custom benchmark listing
 ------------------------
@@ -92,17 +96,17 @@ The benchmark listing file is organised as below.
    # smarts/benchmark/benchmark_listing.yaml
    ---
    benchmarks: # The root element (required)
-     driving_smarts: # The id of the benchmark for reference
-       name: "Driving SMARTS" # The human readable name of the benchmark
+     driving_smarts_2022: # The id of the benchmark for reference
+       name: "Driving SMARTS 2022" # The human readable name of the benchmark
        versions: # A list of benchmark versions
          -
-           # the version of the benchmark, higher is newer
+           # The version of the benchmark, higher is newer
            version: 0.0
-           # the entrypoint for the benchmark, it must have `agent_config`, and `debug_log` as params
+           # The entrypoint for the benchmark, it must have `agent_config`, and `debug_log` as params
            entrypoint: "smarts.benchmark.entrypoints.benchmark_runner_v0.benchmark_from_configs"
-           requirements: ["ray<=2.2.0,>2.0"] # requirements to install if `--auto-install`.
-           params: # additional values to pass into the entrypoint as named keyword arguments.
-             benchmark_config: ${{smarts.benchmark.driving_smarts.v0}}/config.yaml
+           requirements: ["ray<=2.2.0,>2.0"] # Requirements to install if `--auto-install`.
+           params: # Additional values to pass into the entrypoint as named keyword arguments.
+             benchmark_config: ${{smarts.benchmark.driving_smarts.v2022}}/config.yaml
 
 .. note:: 
 

docs/benchmarks/driving_smarts_2022.rst

@@ -3,14 +3,14 @@
 Driving SMARTS 2022
 ===================
 
-The Driving SMARTS 2022 is a benchmark derived from the
+The Driving SMARTS 2022 is a benchmark used in the
 `NeurIPS 2022 Driving SMARTS <https://smarts-project.github.io/archive/2022_nips_driving_smarts/competition/>`_ competition.
 
 Objective
 ---------
 
-Objective is to develop a single policy capable of controlling single-agent or multi-agent to complete different driving scenarios in the ``driving-smarts-competition-v0`` environment.
-Refer to :func:`~smarts.env.gymnasium.driving_smarts_competition_env.driving_smarts_competition_v0_env` for environment details.
+Objective is to develop a single policy capable of controlling single-agent or multi-agent to complete different driving scenarios in the ``driving-smarts-v2022`` environment.
+Refer to :func:`~smarts.env.gymnasium.driving_smarts_2022_env.driving_smarts_2022_env` for environment details.
 
 In each driving scenario, the ego-agents must drive towards their respective mission goal locations. Each agent's mission goal is given in the observation returned by the environment at each time step. 
 The mission goal could be accessed as ``observation.ego_vehicle_state.mission.goal.position`` which gives an ``(x, y, z)`` map coordinate of the goal location.
@@ -72,5 +72,5 @@ See the list of :ref:`available zoo agents <available_zoo_agents>` which are com
     $ cd <path>/SMARTS
     $ scl zoo install <agent path>
     # e.g., scl zoo install zoo/policies/interaction_aware_motion_prediction
-    $ scl benchmark run driving_smarts==0.0 <agent_locator> --auto_install
-    # e.g., scl benchmark run driving_smarts==0.0 zoo.policies:interaction-aware-motion-prediction-agent-v0 --auto-install
+    $ scl benchmark run driving_smarts_2022==0.0 <agent_locator> --auto_install
+    # e.g., scl benchmark run driving_smarts_2022==0.0 zoo.policies:interaction-aware-motion-prediction-agent-v0 --auto-install

smarts/benchmark/benchmark_listing.yaml

@@ -1,13 +1,43 @@
 # smarts/benchmark/benchmark_listing.yaml
 ---
 benchmarks:
-  driving_smarts:
-    name: "Driving SMARTS"
+
+  driving_smarts_2022:
+    name: "Driving SMARTS 2022"
     versions:
       -
         version: 0.0
         entrypoint: "smarts.benchmark.entrypoints.benchmark_runner_v0.benchmark_from_configs"
         requirements: ["ray<=2.2.0,>2.0"]
         params: 
-          benchmark_config: ${{smarts.benchmark.driving_smarts.v0}}/config.yaml
-
+          benchmark_config: ${{smarts.benchmark.driving_smarts.v2022}}/config.yaml
+
+  driving_smarts_2023_1:
+    name: "Driving SMARTS 2023.1"
+    versions:
+      -
+        version: 0.0
+        entrypoint: "smarts.benchmark.entrypoints.benchmark_runner_v0.benchmark_from_configs"
+        requirements: ["ray<=2.2.0,>2.0"]
+        params: 
+          benchmark_config: ${{smarts.benchmark.driving_smarts.v2023}}/config_1.yaml
+
+  driving_smarts_2023_2:
+    name: "Driving SMARTS 2023.2"
+    versions:
+      -
+        version: 0.0
+        entrypoint: "smarts.benchmark.entrypoints.benchmark_runner_v0.benchmark_from_configs"
+        requirements: ["ray<=2.2.0,>2.0"]
+        params: 
+          benchmark_config: ${{smarts.benchmark.driving_smarts.v2023}}/config_2.yaml
+
+  driving_smarts_2023_3:
+    name: "Driving SMARTS 2023.3"
+    versions:
+      -
+        version: 0.0
+        entrypoint: "smarts.benchmark.entrypoints.benchmark_runner_v0.benchmark_from_configs"
+        requirements: ["ray<=2.2.0,>2.0"]
+        params: 
+          benchmark_config: ${{smarts.benchmark.driving_smarts.v2023}}/config_3.yaml

smarts/benchmark/driving_smarts/v0/config.yaml → smarts/benchmark/driving_smarts/v2022/config.yaml

@@ -1,16 +1,16 @@
-# smarts/benchmark/driving_smarts_v0/config.yaml
+# smarts/benchmark/driving_smarts_v2022/config.yaml
 ---
 benchmark:
-  name: "Driving SMARTS V0"
+  name: "Driving SMARTS 2022"
   message: | # A useful message given at the start of the benchmark.
-    This is a cleaned up version of the Driving SMARTS benchmark.
+    This is the Driving SMARTS 2022 benchmark.
 
       Using `TargetPose` agent action has an applied 28m/s cap for agent motion.
       Using `RelativeTargetPose` agent action, the constraint is inbuilt into the action space.
 
       For history see: 
-        - https://codalab.lisn.upsaclay.fr/competitions/6618
         - https://smarts-project.github.io/archive/2022_nips_driving_smarts/
+        - https://codalab.lisn.upsaclay.fr/competitions/6618
   eval_episodes: 50
   debug:
     serial: False
@@ -19,7 +19,7 @@ benchmark:
     headless: true
   envs:
     standard:
-      loc: "smarts.env:driving-smarts-competition-v0"
+      loc: "smarts.env:driving-smarts-v2022"
       scenarios:
         - scenarios/sumo/intersections/1_to_2lane_left_turn_c_agents_1
         - scenarios/sumo/intersections/1_to_2lane_left_turn_t_agents_1
@@ -31,16 +31,16 @@ benchmark:
         - scenarios/sumo/straight/3lane_overtake_agents_1
       kwargs:
         seed: 42
-    # bubble: # reserved for bubble env
-    #   loc: "smarts.env:driving-smarts-competition-bubble-env-v0"
-      # scenarios:
-      #   - 6
-      #   - 4
-      #   - 42
+    # bubble:
+    #   loc: "smarts.env:driving-smarts-bubble-v2022"
+    #   scenarios:
+    #     - 6
+    #     - 4
+    #     - 42
     #   kwargs:
-    # naturalistic: # reserved for driving_smarts_competition_naturalistic-v0
-    #   loc: "smarts.env:driving_smarts_competition_naturalistic-v0"
+    # naturalistic:
+    #   loc: "smarts.env:driving-smarts-naturalistic-v2022"
     #   kwargs:
     #     scenario_dirs:
-    #       - "./scenarios/naturalistic/waymo"
-    #       - "./scenarios/naturalistic/ngsim"
+    #       - scenarios/naturalistic/waymo
+    #       - scenarios/naturalistic/ngsim

smarts/benchmark/driving_smarts/v2023/__init__.py

@@ -0,0 +1,24 @@
+# MIT License
+#
+# Copyright (C) 2023. Huawei Technologies Co., Ltd. All rights reserved.
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+from pathlib import Path
+
+DEFAULT_CONFIG = str((Path(__file__).parent / "config.yaml").absolute())

smarts/benchmark/driving_smarts/v2023/config_1.yaml

@@ -0,0 +1,22 @@
+---
+benchmark:
+  name: "Driving SMARTS 2023.1"
+  message: |
+    This is the Driving SMARTS 2023.1 benchmark.
+
+    For context see: 
+        - https://smarts-project.github.io/competition/2023_driving_smarts/
+        - https://codalab.lisn.upsaclay.fr/competitions/
+  eval_episodes: 50
+  debug:
+    serial: True
+  shared_env_kwargs:
+    seed: 42
+    headless: True
+  envs:
+    standard:
+      loc: "smarts.env:driving-smarts-v2023"
+      scenarios:
+        - scenarios/sumo/straight/3lane_cruise_agents_1
+      kwargs:
+        seed: 42

smarts/benchmark/driving_smarts/v2023/config_2.yaml

@@ -0,0 +1,22 @@
+---
+benchmark:
+  name: "Driving SMARTS 2023.2"
+  message: |
+    This is the Driving SMARTS 2023.2 benchmark.
+
+    For context see: 
+        - https://smarts-project.github.io/competition/2023_driving_smarts/
+        - https://codalab.lisn.upsaclay.fr/competitions/
+  eval_episodes: 50
+  debug:
+    serial: True
+  shared_env_kwargs:
+    seed: 42
+    headless: True
+  envs:
+    turns:
+      loc: "smarts.env:driving-smarts-v2023"
+      scenarios:
+        - scenarios/sumo/intersections/1_to_2lane_left_turn_c_agents_1
+      kwargs:
+        seed: 42

smarts/benchmark/driving_smarts/v2023/config_3.yaml

@@ -0,0 +1,23 @@
+---
+benchmark:
+  name: "Driving SMARTS 2023.3"
+  message: |
+    This is the Driving SMARTS 2023.3 benchmark.
+
+    For context see: 
+        - https://smarts-project.github.io/competition/2023_driving_smarts/
+        - https://codalab.lisn.upsaclay.fr/competitions/
+  eval_episodes: 50
+  debug:
+    serial: True
+  shared_env_kwargs:
+    seed: 42
+    headless: True
+  envs:
+    platoon:
+      loc: "smarts.env:platoon"
+      scenarios:
+        - scenarios/sumo/platoon/straight_sumo_t_agents_1
+      kwargs:
+        seed: 42
+      # metric_formula: smarts/benchmark/driving_smarts/v2023/metric_formula_platoon.py

smarts/core/external_provider.py

@@ -20,12 +20,9 @@
 import weakref
 from typing import Iterable, List, Sequence, Set
 
-import numpy as np
-
 from .actor import ActorRole
 from .controllers import ActionSpaceType
 from .provider import Provider, ProviderManager, ProviderRecoveryFlags, ProviderState
-from .road_map import RoadMap
 from .scenario import Scenario
 from .utils.file import replace
 from .vehicle import VehicleState

smarts/core/plan.py

@@ -330,7 +330,7 @@ def create_route(self, mission: Mission, radius: Optional[float] = None):
                 mission. Defaults to `_default_lane_width` of the underlying
                 road_map.
         """
-        assert not self._route, "already called create_route()"
+        assert not self._route, "Already called create_route()."
         self._mission = mission or Mission.random_endless_mission(self._road_map)
 
         if not self._mission.requires_route:

smarts/core/smarts.py

@@ -55,7 +55,6 @@
 from .scenario import Mission, Scenario
 from .signal_provider import SignalProvider
 from .signals import SignalLightState, SignalState
-from .sumo_traffic_simulation import SumoTrafficSimulation
 from .traffic_history_provider import TrafficHistoryProvider
 from .traffic_provider import TrafficProvider
 from .trap_manager import TrapManager

smarts/core/traffic_provider.py

@@ -19,7 +19,7 @@
 # THE SOFTWARE.
 
 
-from typing import Optional, Sequence
+from typing import Optional
 
 from shapely.geometry import Polygon