tools/sim: Implement `MetaDrive` simulator #27373

jon-chuang · 2023-02-17T11:14:12Z

Description
Add MetaDrive simulator.

Runs at 10 FPS on 1928 X 1208 inputs on an iGPU (Ryzen 6850H - Laptop).

Faster (10 FPS), just the openpilot screen: https://www.youtube.com/watch?v=paVD-laxbcY
Slower (4 FPS), with the render screen: https://www.youtube.com/watch?v=amLu5dloVaI

We disable render screen.

Misc notes:

Refactored bridge.py into clean SimulatorBridge abstraction that shares common logic between CarlaBridge and MetaDriveBridge
bridge.py now accepts --simulator, --ticks_per_frame
Monkey patches MetaDrive methods for improved performance
We have to force WideCameraOnly for performance, in particular, needing to run in ExperimentalMode
We use similar fov=160 as the comma three wide camera & CARLA simulator camera

Verification
Adding an integration test similar to carla simulator's

First part of: #27284

Next steps:

Implement streaming img/obs through Docker? (if openpilot UI already spawns from inside Docker, we don't need to)
More e2e tests (CI: e2e tests with CARLA #26215)

tools/sim/bridge.py

jon-chuang · 2023-02-17T12:01:03Z

tools/sim/bridge.py

    self.params.put_bool("WideCameraOnly", not arguments.dual_camera)
+    self.params.put_bool("DisengageOnAccelerator", True)


Should we add this...?

Hmmm... But anw we can't throttle_out while openpilot selfdrive is running...

adeebshihadeh · 2023-02-17T18:56:47Z

Nice job! Some quick comments until I have some time to properly run and review this:

let's split out all the MetaDrive stuff into its own directory: tools/sim/metadrive/
common stuff shared with CARLA can go into something like tools/sim/common.py, though I wouldn't put much effort into abstractions for CARLA right now
I wouldn't really worry about the full e2e test until this whole thing is super solid, but a simple integration test like CARLA has is definitely welcome

This is my rough plan for openpilot simulator support:

make a simple simulator experience that's extremely reliable (completely ignore CARLA for now)
cleanup openpilot abstractions (e.g. replace the SIMULATOR environment variable checks with something cleaner)
fix up CARLA integration using the nice simulator abstractions from the previous step
do some cool higher-end features in CARLA, like traffic lights and stop signs
use the simulators for e2e tests

We're still in step 1, which I expect will take a bit of time.

tools/sim/bridge.py

tools/sim/bridge/common.py

jon-chuang · 2023-02-18T08:27:46Z

tools/sim/bridge/common.py

-    img = np.frombuffer(image.raw_data, dtype=np.dtype("uint8"))
-    img = np.reshape(img, (H, W, 4))
-    img = img[:, :, [0, 1, 2]].copy()
+  def cam_send_yuv_wide_road(self, yuv):


we don't really need this anymore... (plus there's no point sending an image twice - except maybe to keep openpilot happy? Getting some CommIssue alerts now and again...

jon-chuang · 2023-02-18T09:17:40Z

let's split out all the MetaDrive stuff into its own directory: tools/sim/metadrive/

common stuff shared with CARLA can go into something like tools/sim/common.py, though I wouldn't put much effort into abstractions for CARLA right now

Done, done. CARLA bridge is already factored.

I wouldn't really worry about the full e2e test until this whole thing is super solid, but a simple integration test like CARLA has is definitely welcome

The integration test is not going perfectly on local due to controlsdLagging and commIssue possibly due to my slow GPU. I will test out on my desktop once I get the AIO installed.

cleanup openpilot abstractions (e.g. replace the SIMULATOR environment variable checks with something cleaner)

Could you explain this in greater detail?

My plan:

Regarding MetaDrive simulator:

Next PR implements integration tests, hopefully we can sort out the commIssue or controlsdLagging issues.We will also factor out common test logic into test_helper.py.
Next, we'll make the simulator available as an experimental feature for people to play with on their own machines. We can collect some feedback on:
- performance
- choice of map (the current one has pretty sharp turns and openpilot can fail if turning at too high speeds - see also: Lateral over-correcting in longer/harder corners, riding close or on top of line #26677)
- calibration: accuracy of controls, accuracy of speed display, realtime-ness of the simulation (we should tune for more realtime and be synced up with the rendered display on better devices, but it should be very playable even at low frame rates on low-spec devices like mine).
Once metadrive has released 0.2.7 with their dependencies fixed, we can use that in pyproject.toml (currently uses a patched branch).
Once we are confident, we can enable metadrive by default, rewrite the sim/README.md to emphasize metadrive over CARLA, reorganize scripts, and work on dockerizing.
Concurrently, we should be writing more e2e tests that leverage the capabilities of openpilot. We can start by testing longitudinal control. For metadrive, interactions with other vehicles and with junctions would be nice. At this point, we will enable more features like Radar.

Blocked on:

Issues blocking Integration tests (next PR). Need some assistance on this part.

{"event": "commIssue", "error": true, "invalid": ["liveLocationKalman", "liveParameters", "liveTorqueParameters"], "not_alive": ["testJoystick", "roadCameraState", "driverCameraState"], "not_freq_ok": ["testJoystick", "roadCameraState", "driverCameraState"], "can_rcv_timeout": false}

jon-chuang · 2023-02-18T10:26:29Z

tools/sim/bridge.py

@@ -1,553 +1,24 @@
-#!/usr/bin/env python3


If I rename this file run_bridge.py, the line-by-line changes between bridge.py/common.py would be easier to review. The reviewer can let me know if they prefer this. Then when we merge, we can rename to bridge.py.

renamed to run_bridge.py for now. Let's close this thread once we rename to back to bridge.py

Rmb to do this before merge.

…n/monkey-patch-bridge

tools/sim/bridge/common.py

jon-chuang · 2023-02-18T12:53:19Z

~~Possibly failing due to optional dependencies. (just add pylint)~~ Done.

I'd also like to re-enable the Carla integration test since we've refactored it (probably with errors) and I can't verify locally...

jon-chuang · 2023-03-06T01:31:01Z

didn't really drive out of the box. seems there's an issue with modeld

It was working perfectly at time of submission, and then stopped working so well after merging main a couple of times. I'll debug this.

jon-chuang · 2023-03-06T01:34:27Z

only runs at 10FPS on my machine

You will have to tweak the --ticks_per_frame parameter to possible 1 or 2. The default setting of 10 FPS (--ticks-per-frame=10) was for low-spec devices... (it would be nice to multiplex based on whether a dedicated GPU is detected on the machine).

Part of the bottleneck is running the model with OnnxCpuEP and the other half is the generation of the image data from the simulator.

I'll see if there are any panda3D settings that can help further with the latter.

adeebshihadeh · 2023-03-06T01:44:15Z

@adeebshihadeh did you try mine? https://github.com/MoreTore/openpilot/tree/meta

I haven't. The diff is massive (500+ files) on that branch, so it's not easy to see what I'd be running.

Oh and it was my understanding that the bounty is locked when it's merged.

You're right. Though any after this one will have to be significantly less work to be merged since we're putting in the effort to review the first PR.

If comma clones metadrive I can make a PR to it to add openpilot support.

What do you mean? What needs to go in there that's openpilot specific?

MoreTore · 2023-03-06T01:59:15Z

I haven't. The diff is massive (500+ files) on that branch, so it's not easy to see what I'd be running.

If comma clones metadrive I can make a PR to it to add openpilot support.

I added metadrive to the base dir of openpilot. That was the simplest way for me to develop the changes. It would be added as a submodule in the end. The changes I made include the custom controller, the custom camera adjuster, and the IPC streamer. The streamer increases multi-thread performance.
https://github.com/MoreTore/openpilot/tree/meta/metadrive

…n/monkey-patch-bridge

tools/sim/bridge/metadrive.py

jon-chuang · 2023-03-06T03:37:29Z

The problem for me actually seems to be the CL kernel now. It is sometimes takine 200ms, and 400ms.

I don't recall seeing this initially.

In the worst case, I even get:

KERNEL_PORTION took 4708.432ms
yuv kernel took 4714.306ms

4.7 seconds to run the OpenCL kernel.

It seems that modeld's use of OpenCL is interfering. See for instance:
https://github.com/commaai/openpilot/blob/master/selfdrive/modeld/transforms/transform.h
https://github.com/commaai/openpilot/blob/59ed74ae4a7a51fcfa673393947b755d7b29c323/selfdrive/modeld/models/commonmodel.h

However, I don't understand what is clogging up the queue so badly. It may be some recent changes to master. @adeebshihadeh do you have any idea of recent changes that may have increased the usage of OpenCL kernels by openpilot?

~~Oops, turns out that not running git submodule update --recursive meant that certain errors occured due to miscompiled modules causing some sort of slowdown of opencl somewhere. All good now.~~

Oops, turns out that `LOGPRINT=debug` was actually causing the issue since a voluminous amount of logs is captured and printed on every frame... the observer effect is real!

jon-chuang · 2023-03-06T13:29:37Z

didn't really drive out of the box. seems there's an issue with modeld

This should be fixed now. Just needed to merge master.

only runs at 10FPS on my machine (threadripper + 2080)

Let me know how it goes with --frames_per_tick=2 (50 FPS)

Btw, could you be more specific about what is the base spec we are targetting? Are we targetting laptops with no dGPU (like my current unfortunate setup?)

the output is super spammy, it's hard to see what's going on

Cleaned up the stdout

jon-chuang · 2023-03-06T21:29:58Z

renamed to run_bridge.py for now. Let's close this thread once we rename to back to bridge.py

Reminder

adeebshihadeh · 2023-08-31T21:02:50Z

@jon-chuang we can break up the first stage of the bounty/project, so $500 of the $2k for this PR to be merged with the following goals:

runs comfortably at 20fps on my high-end workstation with a real GPU, i.e. doesn't need to work on a laptop yet
drives within the lane for at least 5s (i.e. it should do something)
simple CI test similar to test_carla_integration.py

Most of the parts are here, just needs to be rebased and cleaned up a bit. I just merged the MetaDrive package in (#29736), so that should reduce the diff a bit.

jon-chuang · 2023-09-06T13:36:19Z

Hello @adeebshihadeh , I'm no longer working on this project @MoreTore can take over if he wants.

adeebshihadeh · 2023-09-06T17:52:29Z

Alright, thanks for kicking this off!

…idge

jnewb1 · 2023-09-21T01:25:54Z

took stuff from this PR and upstreamed it here: #29935

thanks again!

jon-chuang added 4 commits February 17, 2023 19:08

finish

fae33d5

minor

f69eb53

merge-master

26ae348

minor

c42bd64

jon-chuang commented Feb 17, 2023

View reviewed changes

tools/sim/bridge.py Outdated Show resolved Hide resolved

minor

fc3f642

jon-chuang commented Feb 17, 2023

View reviewed changes

jon-chuang added 2 commits February 17, 2023 22:25

poetry

fdfebed

integration test and fix ObdMultiplexing issue

30079a7

jon-chuang commented Feb 17, 2023

View reviewed changes

tools/sim/bridge.py Outdated Show resolved Hide resolved

adeebshihadeh added PC Issues related to running openpilot on PC simulation running openpilot in environments like CARLA labels Feb 18, 2023

jon-chuang added 3 commits February 18, 2023 15:54

refactor

e95c2bf

refactor

142a07e

minor

1d07e5d

jon-chuang commented Feb 18, 2023

View reviewed changes

tools/sim/bridge/common.py Outdated Show resolved Hide resolved

jon-chuang commented Feb 18, 2023

View reviewed changes

tools/sim/bridge/common.py Outdated Show resolved Hide resolved

jon-chuang commented Feb 18, 2023

View reviewed changes

jon-chuang added 2 commits February 18, 2023 16:40

minor

9ce927c

dependencies

1899b41

jon-chuang force-pushed the jon/monkey-patch-bridge branch from 8ffa275 to 1899b41 Compare February 18, 2023 08:41

default simlator is carla

eaff9b3

minor

1d7de6f

jon-chuang commented Feb 18, 2023

View reviewed changes

jon-chuang added 2 commits February 18, 2023 18:30

Merge branch 'master' of https://github.com/commaai/openpilot into jo…

d7afb14

…n/monkey-patch-bridge

rename-for-pr

73c08b4

jon-chuang commented Feb 18, 2023

View reviewed changes

tools/sim/bridge/common.py Outdated Show resolved Hide resolved

jon-chuang added 2 commits March 6, 2023 09:45

minor

d780e22

undo

681ae2c

Merge branch 'master' of https://github.com/commaai/openpilot into jo…

3e492f2

…n/monkey-patch-bridge

jon-chuang commented Mar 6, 2023

View reviewed changes

tools/sim/bridge/metadrive.py Outdated Show resolved Hide resolved

jon-chuang commented Mar 6, 2023

View reviewed changes

tools/sim/bridge/metadrive.py Outdated Show resolved Hide resolved

jon-chuang added 8 commits March 6, 2023 21:34

done

b9123bd

gym dep

cc44ad3

lock

995c1f5

sim bridge unified integration test

a3d1056

empty

086465c

calibrate and complex env

6217b96

rm unnec

f5a83d2

lint

a129edb

adeebshihadeh mentioned this pull request Apr 1, 2023

Refactor bridge.py #26829

Closed

jnewb1 added 2 commits September 7, 2023 13:33

Merge remote-tracking branch 'origin/master' into jon/monkey-patch-br…

b649709

…idge

bump submodules

52ea1c1

jnewb1 mentioned this pull request Sep 13, 2023

Simulator: cleanup in preparation for metadrive #29903

Merged

jnewb1 mentioned this pull request Sep 21, 2023

CI: manage stale PRs #29367

Merged

jnewb1 closed this Sep 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tools/sim: Implement `MetaDrive` simulator #27373

tools/sim: Implement `MetaDrive` simulator #27373

jon-chuang commented Feb 17, 2023 •

edited

Loading

jon-chuang Feb 17, 2023

adeebshihadeh commented Feb 17, 2023 •

edited

Loading

jon-chuang Feb 18, 2023 •

edited

Loading

jon-chuang commented Feb 18, 2023 •

edited

Loading

jon-chuang Feb 18, 2023 •

edited

Loading

jon-chuang Feb 18, 2023 •

edited

Loading

jon-chuang Mar 6, 2023

jon-chuang commented Feb 18, 2023 •

edited

Loading

jon-chuang commented Mar 6, 2023

jon-chuang commented Mar 6, 2023 •

edited

Loading

adeebshihadeh commented Mar 6, 2023

MoreTore commented Mar 6, 2023

jon-chuang commented Mar 6, 2023 •

edited

Loading

jon-chuang commented Mar 6, 2023

jon-chuang commented Mar 6, 2023

adeebshihadeh commented Aug 31, 2023

jon-chuang commented Sep 6, 2023

adeebshihadeh commented Sep 6, 2023

jnewb1 commented Sep 21, 2023

		self.params.put_bool("WideCameraOnly", not arguments.dual_camera)
		self.params.put_bool("DisengageOnAccelerator", True)

tools/sim: Implement MetaDrive simulator #27373

tools/sim: Implement MetaDrive simulator #27373

Conversation

jon-chuang commented Feb 17, 2023 • edited Loading

jon-chuang Feb 17, 2023

Choose a reason for hiding this comment

adeebshihadeh commented Feb 17, 2023 • edited Loading

jon-chuang Feb 18, 2023 • edited Loading

Choose a reason for hiding this comment

jon-chuang commented Feb 18, 2023 • edited Loading

My plan:

Regarding MetaDrive simulator:

jon-chuang Feb 18, 2023 • edited Loading

Choose a reason for hiding this comment

jon-chuang Feb 18, 2023 • edited Loading

Choose a reason for hiding this comment

jon-chuang Mar 6, 2023

Choose a reason for hiding this comment

jon-chuang commented Feb 18, 2023 • edited Loading

jon-chuang commented Mar 6, 2023

jon-chuang commented Mar 6, 2023 • edited Loading

adeebshihadeh commented Mar 6, 2023

MoreTore commented Mar 6, 2023

jon-chuang commented Mar 6, 2023 • edited Loading

jon-chuang commented Mar 6, 2023

jon-chuang commented Mar 6, 2023

adeebshihadeh commented Aug 31, 2023

jon-chuang commented Sep 6, 2023

adeebshihadeh commented Sep 6, 2023

jnewb1 commented Sep 21, 2023

tools/sim: Implement `MetaDrive` simulator #27373

tools/sim: Implement `MetaDrive` simulator #27373

jon-chuang commented Feb 17, 2023 •

edited

Loading

adeebshihadeh commented Feb 17, 2023 •

edited

Loading

jon-chuang Feb 18, 2023 •

edited

Loading

jon-chuang commented Feb 18, 2023 •

edited

Loading

jon-chuang Feb 18, 2023 •

edited

Loading

jon-chuang Feb 18, 2023 •

edited

Loading

jon-chuang commented Feb 18, 2023 •

edited

Loading

jon-chuang commented Mar 6, 2023 •

edited

Loading

jon-chuang commented Mar 6, 2023 •

edited

Loading