Surfaces Revamp

[marc-tonsen]

The surface tracking feature needs a lot of refactoring and a few bug fixes and enhancements.

Progress:

• Does surface tracker take account of camera distortion?  #1260 Compensate camera distortion correctly when mapping gaze onto the surface
• Improve naming in notification API for surfaces
• Refactor code to be more readable / comply with style guides
• Do not block when recomputing surface location cache in offline surface tracker
• Use more than a single image to define a surface in offline surface tracker
• Load surface names and sizes in Player #1261 Load surface names and sizes in Player
• Surface marker are not detected until restart of surface tracker plugin #1247 Surface marker are not detected until restart of surface tracker plugin
• Offline Surface Tracker: Reculation only after finishing add/edit of surface  #1088 Offline Surface Tracker: Recalculation only after finishing add/edit of surface; "Fixed" by making calculations in the background and cancelling/restarting this calculation upon further incoming edits
• recalculate gaze on surfaces in seperate thread #1280 recalculate gaze on surfaces in separate thread
• Improve file naming for exports
• Run export calculations in a background process
• Surface Tracker: Improve heatmap generation #1233 Surface Tracker: Improve heatmap generation
• Get rid of surface id and instead enforce unique names
• UI improvements
• Improved timeline visualizations
• Option to freeze the image in Capture to setup surfaces easier
• Enforce unique surface names

[marc-tonsen]

I pushed a first more or less complete refactor of the Online Surface Tracker.

One issue with surface tracking is camera undistortion. The undistortion function that is defined by the camera intrinsics is only correct inside the bounds of the image. However parts of the surface may lie outside of the image. Given the location of the surface corner outside of the image in undistorted image space, it is impossible for us to correctly compute it's location in distorted image space for visualization. This problem also occurs inside of the image in the very corners.

In the previous version of the surface tracker this problem was not solved, i.e. the visualization was broken if one or more corners of the surface lie outside of the image.
In this version this problem is solved by saving the surface definition in both distorted and undistorted image space. For visualization the version in distorted image space is used, so we never have to undistort. For mapping gaze onto the surface the undistorted version is used to ensure a correct mapping to surface coordinates. The underlying assumption is that the gaze point will always be inside of the image, so it's undistortion is possible.

The refactored version is contained in a new plugin called Surface Tracker Future and the according code is contained in separate module folder in shared_modules. I'd like to receive a first round of feedback / tests before moving on extending this to offline surface tracking.

I droped the 3D localization feature in this version to save time. Since we plan to replace this with a separate and more capable plugin I decided to skip it for now. What do you think about that?

[marc-tonsen]

Currently the homographys output by the surface tracker transform normalized surface coordinates into normalized image coordinates. This makes it difficult to directly use the homographys in e.g. OpenCV to transform an image, which requires the image coordinates to be in pixels. Therefore I propose to switch to transformations between normalized surface coordinates and pixel image coordinates.

[marc-tonsen]

Some of the surface related naming is currently inconsistent/not very descriptive, therefore I propose the following changes. What do you think @papr ?

Goals of renaming:

• make "surf" the consistent acronym for surface
• in filenames always write out surface, no acronym
• remove "screen" from names, as surfaces are not necessarily screens, replace with "surf"
• Make column names consistent across exported files

Changes in Surface Event API:

gaze_on_srf -> gaze_on_surf
fixations_on_srf -> fixations_on_surf
m_to_screen -> surf_to_img_trans
m_from_screen -> img_to_surf_trans

In 'gaze_positions_on_surface_<surface_name>_<surface_id>.csv' file:

world_frame_idx -> world_idx
on_srf -> on_surf

In 'fixations_on_surface_<surface_name>_<surface_id>.csv' file:
on_srf -> on_surf

In 'srf_positions_<surface_name>_<surface_id>.csv' file:

'srf_positions_<surface_name>_<surface_id>.csv' -> 'surface_detections_<surface_name>_<surface_id>.csv'
frame_idx -> world_idx
timestamp -> world_timestamp
m_to_screen -> surf_to_img_trans
m_from_screen -> img_to_surf_trans

In 'surface_events.csv' file:

frame_number -> world_idx
timestamp -> world_timestamp

!!! This file is not part of the surface tracker exports !!!

In 'gaze_positions.csv' file:
index -> world_idx

!!! This file is not part of the surface tracker exports !!!

In 'pupil_positions.csv' file:
index -> world_idx

[papr]

@romanroibu Could you please review the last two commits. Afterwards, I would merge and we can start creating incremental PRs.