Add lockstep mode to video recording #419

iche033 · 2020-10-16T22:53:08Z

This makes sure the ECM updates are lockstepped with GUI camera rendering during video recording so we don't miss frames due to models loading / unloading.

Lockstepping GUI camera and ECM has the side effect that the GUI client may lag behind the server since it can't process the state messages fast enough. I added the ability to configure scene broadcaster system's state publish rate to reduce this problem.

Update: 6533d0d Added support for setting video encoding bitrate in gui.config, and publishing recorder stats to ~/gui/record_video/stats topic

To test recording in lockstep mode, add <lockstep> like so:

<!-- 3D scene -->
<plugin filename="GzScene3D" name="3D View">
  <ignition-gui>
    <title>3D View</title>
    <property type="bool" key="showTitleBar">false</property>
    <property type="string" key="state">docked</property>
  </ignition-gui>

  <engine>ogre2</engine>
  <scene>scene</scene>
  <ambient_light>0.4 0.4 0.4</ambient_light>
  <background_color>0.8 0.8 0.8</background_color>
  <camera_pose>6 0 6 0 0.5 3.14</camera_pose>
  <record_video>
    <use_sim_time>true</use_sim_time>
    <lockstep>true</lockstep>
    <bitrate>8000000</bitrate>
  </record_video>
</plugin>

To configure scene broadcaster's state publish rate, set the <state_hertz> param (default is 60Hz):

    <plugin filename='ignition-gazebo-scene-broadcaster-system'
            name='ignition::gazebo::systems::SceneBroadcaster'>
      <state_hertz>25</state_hertz>
    </plugin>

Signed-off-by: Ian Chen ichen@osrfoundation.org

Signed-off-by: Ian Chen <ichen@osrfoundation.org>

chapulina

I haven't tried it, but besides the global variables, the rest looks reasonable.

src/gui/plugins/scene3d/Scene3D.cc

jennuine

~~Unfortunately no change. Generated videos still have same jumps.~~
I accidentally updated the wrong config file. I changed the correct one, reran the test, and video playback is smooth (no jumps)! 🎉

codecov · 2020-10-16T23:36:56Z

Codecov Report

Merging #419 (3019d67) into ign-gazebo3 (478dea6) will decrease coverage by 0.04%.
The diff coverage is 100.00%.

@@               Coverage Diff               @@
##           ign-gazebo3     #419      +/-   ##
===============================================
- Coverage        77.61%   77.57%   -0.05%     
===============================================
  Files              208      208              
  Lines            11385    11389       +4     
===============================================
- Hits              8837     8835       -2     
- Misses            2548     2554       +6

Impacted Files	Coverage Δ
src/systems/scene_broadcaster/SceneBroadcaster.cc	`97.85% <100.00%> (+0.02%)`	⬆️
src/SimulationRunner.cc	`93.59% <0.00%> (-1.17%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 478dea6...3019d67. Read the comment docs.

Signed-off-by: Ian Chen <ichen@osrfoundation.org>

Add video recorder tutorial

Signed-off-by: Ian Chen <ichen@osrfoundation.org>

chapulina

Works for me. Just have some minor comments. I'll commit the 2 suggestions

src/gui/plugins/scene3d/Scene3D.cc

tutorials/video_recorder.md

src/gui/plugins/scene3d/Scene3D.cc

Signed-off-by: Louise Poubel <louise@openrobotics.org>

* Clarify how sim time is interpreted in a System's step (#467) * add frame_id and child_frame_id attribute support for DiffDrive (#361) * Add ability to record video based on sim time (#414) * add ability to record video from gui camera using sim time * add msg * use QueryBoolText * Add lockstep mode to video recording (#419) * Disable right click menu when using measuring tool (#458) * Bump to 3.6.0 (#524) * Don't make docs on macOS (#528) * Updates to ardupilot migration tutorial (#525) * Update gtest to 1.10.0 for Windows compilation (ign-gazebo3) (#506) * Compile new gtest with c++11 * Use INSTANTIATE_TEST_SUITE_P instead of deprecated -INSTANTIATE_TEST_CASE_P * Apply suggestions from code review * One more tutorial version bump * Fix bad merge * change nullptr to a int ptr for qt 5.15.2 bug (#527) * Generate valid topics everywhere (support names with spaces) (#522) * Change deprecated test case->suite Co-authored-by: Ashton Larkin <ashton@openrobotics.org> Co-authored-by: G.Doisy <doisyg@users.noreply.github.com> Co-authored-by: Ian Chen <ichen@osrfoundation.org> Co-authored-by: Nate Koenig <nate@openrobotics.org> Co-authored-by: Louise Poubel <louise@openrobotics.org> Co-authored-by: John Shepherd <john@openrobotics.org> Co-authored-by: Steve Peters <scpeters@openrobotics.org> Co-authored-by: Jose Luis Rivero <jrivero@osrfoundation.org> Co-authored-by: Akash Patel <17132214+acxz@users.noreply.github.com>

* Initial commit Signed-off-by: Martiño Crespo <marticres@gmail.com> * Added plugin files Signed-off-by: Martiño Crespo <marticres@gmail.com> * Initial .sdf file Signed-off-by: Martiño Crespo <marticres@gmail.com> * Pipeline for getting sensor contacts Signed-off-by: Martiño Crespo <marticres@gmail.com> * Added visualization of postion and forces Signed-off-by: Martiño Crespo <marticres@gmail.com> * Added plugin for Transform Control Signed-off-by: Martiño Crespo <marticres@gmail.com> * Added initial interpolation Signed-off-by: Martiño Crespo <marticres@gmail.com> * Code check Signed-off-by: Martiño Crespo <marticres@gmail.com> * Update info after specified milliseconds Signed-off-by: Martiño Crespo <marticres@gmail.com> * Set marker lifetime for better performance Signed-off-by: Martiño Crespo <marticres@gmail.com> * Added Depth Camera Signed-off-by: Martiño Crespo <marticres@gmail.com> * Added callback and unpacking to Depth Camera messages Signed-off-by: Martiño Crespo <marticres@gmail.com> * Compute and visualize normal forces Signed-off-by: Martiño Crespo <marticres@gmail.com> * PR Feedback 1 Signed-off-by: Martiño Crespo <marticres@gmail.com> * Update sdf example to a more realistic environment Signed-off-by: Martiño Crespo <marticres@gmail.com> * Allow moving Depth Camera from model origin Signed-off-by: Martiño Crespo <marticres@gmail.com> * Make markers dimensions available as parameters Signed-off-by: Martiño Crespo <marticres@gmail.com> * Visualize sensor as a marker instead of <visual> and filter out normal forces outside sensor Signed-off-by: Martiño Crespo <marticres@gmail.com> * PR Feedback 2 Signed-off-by: Martiño Crespo <marticres@gmail.com> * PR Feedback 3 Signed-off-by: Martiño Crespo <marticres@gmail.com> * Minor fixes for sdf, sensor marker and profiler Signed-off-by: Martiño Crespo <marticres@gmail.com> * PR Feedback 4 Signed-off-by: Martiño Crespo <marticres@gmail.com> * fix mac warning Signed-off-by: Mabel Zhang <mabel@openrobotics.org> * Helper function to set component data (gazebosim#436) Signed-off-by: Louise Poubel <louise@openrobotics.org> * Remove unneeded if statement (gazebosim#432) Signed-off-by: John Shepherd <john@openrobotics.org> * Fixes flaky RecordAndPlayback test in INTEGRATION_log_system (gazebosim#463) The flakiness comes from two sources: 1. Poses recorded by the LogRecorder are published by the SceneBroadcaster system throttled at 60 Hz. The throttle mechanism uses real-time instead of sim-time which causes a variance in the number of recorded poses from run to run. However, the expected number of recorded poses was calculated with the assumption that the simulation would run with a 1.0 RTF. If the CPU load is high, there could be a mismatch between the expected and the actual number of recorded poses, which causes the test to fail. This can be checked by running the test with `cpulimit -l 20 -f bin/INTEGRATION_log_system -- --gtest_filter="*RecordAndPlayback"` 2. An attempt is made to match up played back poses with the closest timestamp in the recorded file. These poses are again published by the SceneBroadcaster, so they are subject to the same kind of timing issues as the recorded poses. The solution in this patch for the first issue is to determine the expected number of recorded poses by counting them in a separate system that mimics the throttling mechanism of ign-transport. For the second issue, a testing system is added to the playback server bypassing the SceSceneBroadcaster altogether. Signed-off-by: Addisu Z. Taddese <addisu@openrobotics.org> * Make PeerTracker test more robust (gazebosim#452) Signed-off-by: Louise Poubel <louise@openrobotics.org> Co-authored-by: Michael Carroll <michael@openrobotics.org> * Clarify how sim time is interpreted in a System's step (gazebosim#467) Signed-off-by: Ashton Larkin <ashton@openrobotics.org> * 3 ➡️ 4: fixes for gazebosim#463 (gazebosim#469) Signed-off-by: Louise Poubel <louise@openrobotics.org> * Link link tutrial (gazebosim#472) Signed-off-by: Louise Poubel <louise@openrobotics.org> * Switch to async state service request (gazebosim#461) Signed-off-by: Ian Chen <ichen@osrfoundation.org> * Use a std::promise/std::future to avoid busy waiting the step ack messages in NetworkManagerPrimary (gazebosim#470) Signed-off-by: Ivan Santiago Paunovic <ivanpauno@ekumenlabs.com> Signed-off-by: Louise Poubel <louise@openrobotics.org> Co-authored-by: Louise Poubel <louise@openrobotics.org> * Fix tests that use Fuel paths with uppercase letters (gazebosim#480) As of gazebosim/gz-fuel-tools#130, fuel paths use all lowercase letters. This fixes some tests that had uppercase letters. Signed-off-by: Addisu Z. Taddese <addisu@openrobotics.org> * 4.1.0 (gazebosim#485) Signed-off-by: Ashton Larkin <ashton@openrobotics.org> * Update key event handling (gazebosim#466) Signed-off-by: John Shepherd <john@openrobotics.org> * Fix slot in Plotting plugin (gazebosim#490) Signed-off-by: Alejandro Hernández <ahcorde@gmail.com> * Tape Measure Plugin (gazebosim#456) Signed-off-by: John Shepherd <john@openrobotics.org> * Satisfy make codecheck (gazebosim#491) Signed-off-by: Louise Poubel <louise@openrobotics.org> * Move deselect and preview termination to render thread (gazebosim#493) Signed-off-by: John Shepherd <john@openrobotics.org> * Fix codecheck (gazebosim#499) Signed-off-by: Nate Koenig <nate@openrobotics.org> Co-authored-by: Nate Koenig <nate@openrobotics.org> * Logical Audio Sensor Plugin (gazebosim#401) Signed-off-by: Ashton Larkin <ashton@openrobotics.org> * Codecheck - initialize primitive in LogicalAudio component (gazebosim#502) Signed-off-by: Ashton Larkin <ashton@openrobotics.org> * 3 to 4: fix codecheck for ign-gazebo4 Signed-off-by: Ashton Larkin <ashton@openrobotics.org> * 3 to 4: resolve codecheck warnings Signed-off-by: Ashton Larkin <ashton@openrobotics.org> * add frame_id and child_frame_id attribute support for DiffDrive (gazebosim#361) Add configuration of the odom frame_id and child_frame_id fields from sdf attributes <frame_id> and <child_frame_id> Signed-off-by: Guillaume <guillaume.doisy@wyca.fr> Signed-off-by: Guillaume Doisy <guillaume.doisy@wyca.fr> * Require ign-gui 4.1.0 (gazebosim#505) Signed-off-by: Louise Poubel <louise@openrobotics.org> * Fix shadow artifacts by disabling double sided rendering (gazebosim#446) * read double sided sdf param Signed-off-by: Ian Chen <ichen@osrfoundation.org> * update migration Signed-off-by: Ian Chen <ichen@osrfoundation.org> * Make the tunnels example world more interesting (gazebosim#462) Signed-off-by: Louise Poubel <louise@openrobotics.org> * add double sided msg to sdf conversion Signed-off-by: Ian Chen <ichen@osrfoundation.org> Co-authored-by: Louise Poubel <louise@openrobotics.org> * Add ability to record video based on sim time (gazebosim#414) * add ability to record video from gui camera using sim time Signed-off-by: Ian Chen <ichen@osrfoundation.org> * add msg Signed-off-by: Ian Chen <ichen@osrfoundation.org> * use QueryBoolText Signed-off-by: Ian Chen <ichen@osrfoundation.org> Co-authored-by: Nate Koenig <nate@openrobotics.org> Co-authored-by: Louise Poubel <louise@openrobotics.org> * Add lockstep mode to video recording (gazebosim#419) Signed-off-by: Ian Chen <ichen@osrfoundation.org> Signed-off-by: Louise Poubel <louise@openrobotics.org> Co-authored-by: Nate Koenig <nate@openrobotics.org> Co-authored-by: Louise Poubel <louise@openrobotics.org> * Kinetic energy monitor plugin (gazebosim#492) Signed-off-by: Gonzalo de Pedro <gonzalo@depedro.com.ar> Signed-off-by: Nate Koenig <nate@openrobotics.org> Co-authored-by: Nate Koenig <nate@openrobotics.org> Co-authored-by: Louise Poubel <louise@openrobotics.org> * Disable right click menu when using measuring tool (gazebosim#458) Signed-off-by: John Shepherd <john@openrobotics.org> Co-authored-by: Louise Poubel <louise@openrobotics.org> * Bump to 3.6.0 (gazebosim#524) Signed-off-by: Louise Poubel <louise@openrobotics.org> * Don't make docs on macOS (gazebosim#528) add comment about doxygen bug Signed-off-by: Louise Poubel <louise@openrobotics.org> Signed-off-by: Steve Peters <scpeters@openrobotics.org> Co-authored-by: Louise Poubel <louise@openrobotics.org> * Updates to ardupilot migration tutorial (gazebosim#525) Signed-off-by: Louise Poubel <louise@openrobotics.org> * Update gtest to 1.10.0 for Windows compilation (ign-gazebo3) (gazebosim#506) * Compile new gtest with c++11 * Use INSTANTIATE_TEST_SUITE_P instead of deprecated -INSTANTIATE_TEST_CASE_P Signed-off-by: Jose Luis Rivero <jrivero@osrfoundation.org> * Apply suggestions from code review Signed-off-by: Michael Carroll <michael@openrobotics.org> Co-authored-by: Louise Poubel <louise@openrobotics.org> * change nullptr to a int ptr for qt 5.15.2 bug (gazebosim#527) See: https://bugreports.qt.io/browse/QTBUG-89114 Signed-off-by: acxz <17132214+acxz@users.noreply.github.com> Co-authored-by: Louise Poubel <louise@openrobotics.org> Signed-off-by: Louise Poubel <louise@openrobotics.org> * Generate valid topics everywhere (support names with spaces) (gazebosim#522) Signed-off-by: Louise Poubel <louise@openrobotics.org> * One more tutorial version bump Signed-off-by: Michael Carroll <michael@openrobotics.org> * Fix bad merge Signed-off-by: Michael Carroll <michael@openrobotics.org> * change nullptr to a int ptr for qt 5.15.2 bug (gazebosim#527) See: https://bugreports.qt.io/browse/QTBUG-89114 Signed-off-by: acxz <17132214+acxz@users.noreply.github.com> Co-authored-by: Louise Poubel <louise@openrobotics.org> Signed-off-by: Louise Poubel <louise@openrobotics.org> * Generate valid topics everywhere (support names with spaces) (gazebosim#522) Signed-off-by: Louise Poubel <louise@openrobotics.org> * Change deprecated test case->suite Signed-off-by: Michael Carroll <michael@openrobotics.org> * 3 ➡️ 4 (gazebosim#533) * Clarify how sim time is interpreted in a System's step (gazebosim#467) * add frame_id and child_frame_id attribute support for DiffDrive (gazebosim#361) * Add ability to record video based on sim time (gazebosim#414) * add ability to record video from gui camera using sim time * add msg * use QueryBoolText * Add lockstep mode to video recording (gazebosim#419) * Disable right click menu when using measuring tool (gazebosim#458) * Bump to 3.6.0 (gazebosim#524) * Don't make docs on macOS (gazebosim#528) * Updates to ardupilot migration tutorial (gazebosim#525) * Update gtest to 1.10.0 for Windows compilation (ign-gazebo3) (gazebosim#506) * Compile new gtest with c++11 * Use INSTANTIATE_TEST_SUITE_P instead of deprecated -INSTANTIATE_TEST_CASE_P * Apply suggestions from code review * One more tutorial version bump * Fix bad merge * change nullptr to a int ptr for qt 5.15.2 bug (gazebosim#527) * Generate valid topics everywhere (support names with spaces) (gazebosim#522) * Change deprecated test case->suite Co-authored-by: Ashton Larkin <ashton@openrobotics.org> Co-authored-by: G.Doisy <doisyg@users.noreply.github.com> Co-authored-by: Ian Chen <ichen@osrfoundation.org> Co-authored-by: Nate Koenig <nate@openrobotics.org> Co-authored-by: Louise Poubel <louise@openrobotics.org> Co-authored-by: John Shepherd <john@openrobotics.org> Co-authored-by: Steve Peters <scpeters@openrobotics.org> Co-authored-by: Jose Luis Rivero <jrivero@osrfoundation.org> Co-authored-by: Akash Patel <17132214+acxz@users.noreply.github.com> * Add support for topic statistics on breadcrumb deployments (gazebosim#532) * Add support for topic statistics on breadcrumb deployments Signed-off-by: Nate Koenig <nate@openrobotics.org> * Require version 9.1 of ignition transport Signed-off-by: Nate Koenig <nate@openrobotics.org> * Move to after mutex Signed-off-by: Nate Koenig <nate@openrobotics.org> Co-authored-by: Nate Koenig <nate@openrobotics.org> Co-authored-by: Carlos Agüero <caguero@osrfoundation.org> Co-authored-by: Martiño Crespo <marticres@gmail.com> Co-authored-by: Louise Poubel <louise@openrobotics.org> Co-authored-by: Mabel Zhang <mabel@openrobotics.org> Co-authored-by: John Shepherd <john@openrobotics.org> Co-authored-by: Addisu Z. Taddese <addisu@openrobotics.org> Co-authored-by: Michael Carroll <michael@openrobotics.org> Co-authored-by: Ashton Larkin <ashton@openrobotics.org> Co-authored-by: Ian Chen <ichen@osrfoundation.org> Co-authored-by: Ivan Santiago Paunovic <ivanpauno@ekumenlabs.com> Co-authored-by: Alejandro Hernández Cordero <ahcorde@gmail.com> Co-authored-by: Nate Koenig <nkoenig@users.noreply.github.com> Co-authored-by: Nate Koenig <nate@openrobotics.org> Co-authored-by: G.Doisy <doisyg@users.noreply.github.com> Co-authored-by: Gonzo <42421541+gonzodepedro@users.noreply.github.com> Co-authored-by: Steve Peters <scpeters@openrobotics.org> Co-authored-by: Jose Luis Rivero <jrivero@osrfoundation.org> Co-authored-by: Akash Patel <17132214+acxz@users.noreply.github.com> Co-authored-by: Carlos Agüero <caguero@osrfoundation.org>

iche033 and others added 4 commits October 15, 2020 17:34

add ability to record video from gui camera using sim time

cb7d854

Signed-off-by: Ian Chen <ichen@osrfoundation.org>

add msg

a12d948

Signed-off-by: Ian Chen <ichen@osrfoundation.org>

Merge branch 'ign-gazebo3' into record_sim_time

9c6da96

add lockstep mode during video recording

925d229

Signed-off-by: Ian Chen <ichen@osrfoundation.org>

iche033 requested review from nkoenig and jennuine October 16, 2020 22:53

iche033 requested a review from chapulina as a code owner October 16, 2020 22:53

chapulina reviewed Oct 16, 2020

View reviewed changes

src/gui/plugins/scene3d/Scene3D.cc Outdated Show resolved Hide resolved

jennuine reviewed Oct 16, 2020

View reviewed changes

add support for configurating state pub rate

7348bf4

Signed-off-by: Ian Chen <ichen@osrfoundation.org>

jennuine approved these changes Oct 17, 2020

View reviewed changes

iche033 mentioned this pull request Oct 17, 2020

Fix video encoder timing gazebosim/gz-common#105

Merged

chapulina assigned iche033 Oct 19, 2020

iche033 added 2 commits October 20, 2020 23:48

set bit rate and pub recorder video stats

6533d0d

Signed-off-by: Ian Chen <ichen@osrfoundation.org>

strict lockstepping with gui camera

90f78fe

Signed-off-by: Ian Chen <ichen@osrfoundation.org>

This was referenced Oct 26, 2020

Add playback event recorder osrf/subt#633

Merged

Video recordings are not really nice to view gazebosim/gz-common#109

Open

iche033 added 4 commits November 2, 2020 16:01

add video recorder tutorial

7dccc0a

Signed-off-by: Ian Chen <ichen@osrfoundation.org>

add gif

31619cd

Signed-off-by: Ian Chen <ichen@osrfoundation.org>

add giff

d19f507

Signed-off-by: Ian Chen <ichen@osrfoundation.org>

updates to gui configurations

f3ee42c

Signed-off-by: Ian Chen <ichen@osrfoundation.org>

iche033 mentioned this pull request Nov 3, 2020

Add video recorder tutorial #438

Merged

iche033 added 4 commits November 2, 2020 18:01

update link, typos

46a7d1f

Signed-off-by: Ian Chen <ichen@osrfoundation.org>

add warning msg if lockstep=true and sim_time=false

aa6f031

Signed-off-by: Ian Chen <ichen@osrfoundation.org>

updates based on feedback

bdee780

Signed-off-by: Ian Chen <ichen@osrfoundation.org>

Merge pull request #438 from ignitionrobotics/video_recorder_tutorial

e69fd35

Add video recorder tutorial

iche033 requested a review from maryaB-osr as a code owner November 17, 2020 00:34

iche033 added 3 commits December 1, 2020 13:38

Merge branch 'ign-gazebo3' into record_sim_time

df70901

remove a few global var

d8e57c6

Signed-off-by: Ian Chen <ichen@osrfoundation.org>

merge from ign-gazebo3

c08f389

Signed-off-by: Ian Chen <ichen@osrfoundation.org>

iche033 requested a review from azeey as a code owner December 2, 2020 00:09

Merge branch 'record_sim_time' into record_lockstep

c331fe4

iche033 removed the request for review from azeey December 2, 2020 00:13

iche033 mentioned this pull request Dec 11, 2020

Add ability to record video based on sim time #414

Merged

nkoenig approved these changes Dec 16, 2020

View reviewed changes

chapulina added the 🏰 citadel Ignition Citadel label Dec 18, 2020

Base automatically changed from record_sim_time to ign-gazebo3 December 21, 2020 21:04

iche033 added 2 commits December 21, 2020 13:22

merge from ign-gazebo3

1c6d23a

Signed-off-by: Ian Chen <ichen@osrfoundation.org>

resolve conflicts

154139b

Signed-off-by: Ian Chen <ichen@osrfoundation.org>

chapulina approved these changes Dec 22, 2020

View reviewed changes

src/gui/plugins/scene3d/Scene3D.cc Outdated Show resolved Hide resolved

tutorials/video_recorder.md Show resolved Hide resolved

src/gui/plugins/scene3d/Scene3D.cc Outdated Show resolved Hide resolved

Fix clang warning, typo

3019d67

Signed-off-by: Louise Poubel <louise@openrobotics.org>

chapulina merged commit bea0879 into ign-gazebo3 Dec 23, 2020

chapulina deleted the record_lockstep branch December 23, 2020 17:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add lockstep mode to video recording #419

Add lockstep mode to video recording #419

iche033 commented Oct 16, 2020 •

edited

Loading

chapulina left a comment

jennuine left a comment •

edited

Loading

codecov bot commented Oct 16, 2020 •

edited

Loading

chapulina left a comment

Add lockstep mode to video recording #419

Add lockstep mode to video recording #419

Conversation

iche033 commented Oct 16, 2020 • edited Loading

chapulina left a comment

Choose a reason for hiding this comment

jennuine left a comment • edited Loading

Choose a reason for hiding this comment

codecov bot commented Oct 16, 2020 • edited Loading

Codecov Report

chapulina left a comment

Choose a reason for hiding this comment

iche033 commented Oct 16, 2020 •

edited

Loading

jennuine left a comment •

edited

Loading

codecov bot commented Oct 16, 2020 •

edited

Loading