Kinefly
----------------
Kinefly extracts kinematics variables from the camera image of a tethered winged insect.

You may track the movement of each bodypart:  head, abdomen, leftwing, rightwing.  Several tracking techniques are available to choose from (area tracking, edge finding, tip tracking).  An auxillary elliptical region may also be monitored (intended to capture leg motion or wingbeat frequency). 

The software runs on ROS, the Robot Operating System, and can use any camera or other source that provides a ROS image stream, e.g. /camera/image_raw.



Contents:
---------------------------
Installation
Running
Main Window
Handle Positioning
Output Variables
Units
Bodypart Trackers
 - Area Tracking
 - Edge Tracking
 - Tip Tracking
 - Intensity Tracking and Wingbeat Frequency
Frame Rate
White on Black
Background Subtraction
Parameters
Voltage Output
Record Video for Later
Retrack Prerecorded Video
LED Panels
ROS Commands
Plot Tracker (auxillary program to view tracking performance)



Installation:
---------------------------
You need:
* ROS
* camera to supply a ROS image stream
* Kinefly (at github.com/ssafarik/Kinefly)
* ledpanels (at github.com/ssafarik/ledpanels)
* phidgets (at github.com/ssafarik/phidgets)  

The Kinefly code is independent of the specific hardware and settings that you're using.  This is accomplished by using parameters that are specific to your hardware (i.e. RIG).  The parameters are set in a subdirectory of the Kinefly/launch directory, and each machine sets an environment variable 'RIG' that tells Kinefly which set of parameters to use.
 
If you want to output voltages (e.g. voltages to indicate wing angles, etc), then you'll need the PhidgetsAnalog hardware w/ Phidgets API.  If you want to have the Kinefly measurements control LED panels, then you'll need panels hardware, see below.  There might be more stuff, but that's the basic setup.

Refer to the file 'install.txt' for detailed instructions.


Running:
---------------------------
To run Kinefly, you first need to ensure that roscore is running.  In a terminal window (note below that the $ is the command prompt, not typed by you):
$ roscore

Next, depending on if you have an ethernet or firewire or USB camera, edit the Kinefly/launch/RIG/source_camera.launch file as appropriate (where 'RIG' is the name of your rig).  You may also need to edit the 'params_camera.launch' or 'params_kinefly.launch' files to set exposure, framerate, tracking params, etc.  

Then in another terminal window use the roslaunch command to launch Kinefly:
$ roslaunch Kinefly main.launch



Main Window
---------------------------
The main window provides an interface to most, but not all, of the functionality. The buttons and checkboxes on the toolbar, and the handles on the image, function as follows:
  <Exit>        Terminate the program.  Use this instead of the 
                titlebar's [x].
  <SaveBG>      Save the instant camera image to disk as ~/kinefly.png
  Track:
  []H           Turn on/off head tracking.
  []A           Turn on/off abdomen tracking.
  []L           Turn on/off left wing tracking.
  []R           Turn on/off right wing tracking.
  []X           Turn on/off the 'aux' area intensity & wingbeat frequency
                calculations.
  Subtract:
  []H           Turn on/off background subtraction on head
  []A           Turn on/off background subtraction on abdomen
  []LR          Turn on/off background subtraction on left & right wings
  []X           Turn on/off background subtraction on aux
  []Stabilize   Turn on/off image stabilization for head & abdomen.  The  
                quality of stabilization is very dependent upon the region 
                selected with the handles. (default: false) 
  []Symmetric   When moving handles, keep them symmetric about the body axis.
  []Windows     Turn on/off helpful windows showing various internal images.

  
Handle Positioning:
---------------------------
Each of the head/abdomen/left/right body parts has a set of control points that determine what part of the image is tracked.  The handle positions determine the hinges, angles, and radii.

First, for best tracking, each of the head/abdomen/left/right body parts should have the 'hinge' handle placed at the center of rotation, which for example in the case of the abdomen is not the center of the abdomen, but rather is the point where the abdomen connects to the rest of the body, and rotates about.

For the head and abdomen, set the inner and outer radii to cover just a narrow strip that encloses unique features of the bodypart as the bodypart rotates, and set the angles to cover all of the bodypart motion.  Reliable head rotation seems to be the most dependent on proper handle positions.  

For the wings, the angles and radii should be positioned to indicate a region of interest such that legs and other non-wing moving things do not encroach into the region of interest.


Output Variables
---------------------------
The output is published on the ROS topic '/kinefly/flystate', and can be retrieved either by running the command 'rostopic echo kinefly/flystate' or by writing your own ROS node and subscribing to that topic.

The fields shown in [brackets] are lists of values, and there may be zero, one, or more than one entry (e.g. when finding multiple wing edges).  Which fields contain valid values are dependent upon which bodypart tracker has been set, and its settings.  

An example of the flystate output is shown here:
---
header: 
  seq: 1033
  stamp: 
    secs: 1393885525
    nsecs: 419325113
  frame_id: Fly
head: 
  angles: [-0.05834060178390743]
  gradients: []
  radii: [54.635968488535795]
  freq: 0.0
  intensity: 0.45413453997
abdomen: 
  angles: [-0.37292074013475407]
  gradients: []
  radii: [116.38383782940254]
  freq: 0.0
  intensity: 0.591699666712
left: 
  angles: [0.8233633148061248]
  gradients: [-0.00784313678741455]
  radii: []
  freq: 0.0
  intensity: 0.802197675672
right: 
  angles: [0.6048098145857046]
  gradients: [0.011551618576049805]
  radii: []
  freq: 0.0
  intensity: 0.593512067496
aux: 
  angles: []
  gradients: []
  radii: []
  freq: 0.0
  intensity: 0.673986792564
 

Units
---------------------------
Angle units are radians.  
Radius units are pixels.  
Intensity is on the range [0,1], where 0.0 equals all pixels black, and 1.0 equals all pixels white.  The main window shows intensity units in parentheses, ().
Gradient is the intensity change per unit, and is used for edge and tip finding.  Edge-tracking gradient is change per angle, and tip-tracking gradient is change per radial distance.

Each bodypart (head, abdomen, and wings) has its own angle origin, which in general terms is outward from the body center.  The zero angle for the head is directly opposite from the abdomen, and the zero of the abdomen angle is directly opposite from the head.  The zero for each wing is directly opposite from the other wing hinge.  In other words, imagine a line connecting the head and abdomen hinges, and a line connecting each wing hinge.  The zero angles are outward along those lines, and the range of each angle is on the range [-pi,+pi].


Bodypart Trackers:
---------------------------
The trackers available are as follows, and are specified via the 'tracker' parameter in the params_kinefly.launch file.  Any bodypart (except aux) can use any tracker.
'area'      Uses image registration techniques, and reports the rotation and
            radial movement of the area as a whole.
'edge'      Finds radial edges, ordered by decreasing edge strength.
'tip'       Finds the point where the bodypart is farthest from
            the rotation center.  
'intensity' Monitor the pixel intensity, and the frequency of periodic 
            intensity, of an elliptical area

There is an auxillary Plot Tracker tool that may be helpful for setting wing edge detection thresholds, etc.  See the Plot Tracker section of this document for details.


Area Tracking:
---------------------------
This is the default for the head & abdomen.  We measure rotation and radial expansion of the bodypart via an image registration technique in polar coordinates on the region of interest.  This state measurement is autozero'ed such that the origin tends to be near the midpoint of the range of motion.  Autoranging will therefore cause occasional changes of the angle origin, making it seem as though there were a step change in the angle when there wasn't.  Autoranging can be turned off by changing the parameter setting.  

Area tracking is also affected by a "feathering" (i.e. a window function) parameter that smooths the edge pixels of the bodypart.  Setting feathering=0 gives a rectangular window (i.e. a hard edge), and feathering=1 gives a Hanning window (i.e. a very smooth edge).  Tracking of small body parts, such as a fly head, may benefit from setting feathering=0.  The default is 0.25.


Edge Tracking:
---------------------------
This is the default for left & right wings.  We find multiple edge angles per wing (you can use this feature to track legs etc too, if desired), the number of edges being specified via parameter 'n_edges_max'.  The edge angles are in one list, and their corresponding intensity gradients are in another.  The edges are found by looking at the intensity gradient across the wingstroke, and each list is sorted in descending order of absolute gradient.

The parameter 'threshold' can be used to reduce false detections when no edge is present, and will typically be a low value such as 0.01.  It is set independently for each wing.     


Tip Tracking:
---------------------------
When the 'tip' tracker is set for a bodypart, We locate the bodypart point farthest from the point of rotation.  The parameter 'threshold' is compared to the radial gradient to find the bodypart, and should be set to approx 0.01. 


Intensity Tracking and Wingbeat Frequency:
---------------------------
We make an attempt to measure the wingbeat frequency using the frequency spectrum of the aux intensity.  For this to work, several criteria must be met.  First, you must set the valid range of wingbeat frequencies in the params_kinefly.launch file.  Upon the next launch, Kinefly will print out the compatible camera framerate needed (the lowest typically being slightly higher than twice the wingbeat passband width).  Next, set the camera to that framerate.  If the camera framerate can be maintained within the valid range, then Kinefly will compute the wingbeat frequency.  Since the framerate is typically much less than the Nyquist frequency needed to cover the highest wingbeat frequency, we use an undersampling technique to sample just the passband, and look at a baseband alias of the passband frequencies.  Proper camera framerate is critical to the functioning of this feature, and the computer must be fast enough to process the frames in the allowed time, with no camera framerate variation outside of the allowed range.  If you have trouble maintaining the framerate, consider using the parameter 'scale_image'.      

  
Frame Rate
---------------------------
The frame rate is dependent upon virtually every setting.  Turn off tracking of unneeded bodyparts, use the smallest usable regions of interest (i.e. handle positions), and turn off unneeded windows to speed the frame rate.  The largest improvement is likely to come from lowering the value of 'scale_image' in the params_kinefly.launch file.

  
White On Black
---------------------------
Kinefly works best internally if the image is white-on-black.  The camera image may be either black-on-white or white-on-black, and whether to invert the color or not is decided automatically based on the pixel intensity of a region centered on the intersection of the lines head/abdomen hinges and left/right hinges.  If the mean pixel value of this region is darker than the mean pixel value of the overall image, then we invert the colors.

The user can override the autodetect and set the behavior manually via the checkbox on the toolbar.  When Kinefly launches it starts in autodetect mode, but once the checkbox is clicked, then the manual setting is used until the next launch of Kinefly.  The setting is not saved across launches.


Background Subtraction
---------------------------
Background subtraction can be turned on/off for each bodypart individually.  The background is initialized with the kinefly.png image on disk, and updates as a moving average with the time constant (in seconds) parameter rc_background. 

Saturation of the background image can cause problems with tracking, due to limited foreground headroom.  Kinefly contains a "saturation correction" algorithm to reduce or eliminate this effect, however the feature is turned off by default due to the computational cost and reduced framerate.  You may turn it on/off for each bodypart separately via the parameter "saturation_correction".  The feature works by reducing the importance of pixels the closer they are to saturated.      


Parameters.
---------------------------
Parameters are stored in the file 'Kinefly/launch/params_kinefly.launch'.  

Anything set via the GUI (e.g. handle positions, checkbox states, etc) are automatically saved to the kinefly.yaml file.

The non-gui parameters are below.  

General Parameters:
    filenameBackground:         Location to store the background image.  
                                (default: ~/kinefly.png)
    image_topic:                ROS topic for the image stream.  Running
                                the ROS node image_proc in addition to
                                Kinefly and setting this to image_mono
                                may help with some cameras that provide
                                bayer encoded images.
                                (default: /camera/image_raw)
    n_queue_images:             Length of the circular buffer for incoming
                                images. (default: 2)
    n_edges_max:                Max number of wing edges to find. (default: 1)
    rc_background:              Time constant of the moving average background
                                image. (default: 1000.0)
    scale_image:                Scale all images by this amount.  Smaller 
                                values give better framerate. (default: 1.0)
    use_gui:                    Turn on/off the graphical user interface to
                                speed the framerate. (default: true)
    parameterfile:              Location of the .yaml file for keeping track
                                of GUI handle positions & checkboxes. 
                                (default: ~/kinefly.yaml)
  
Head/Abdomen/Left/Right Parameters:
    tracker:                    Which feature to track.  Options are 'area',
                                'edge', and 'tip'. 
                                (default: area/area/edge/edge)
    saturation_correction:      Correct for bright pixels in the background
                                image to improve tracking. (default: false)
                                
Parameters that apply to Area tracking:
    autozero:                   Automatically (true) determine the zero 
                                angle based on center of motion, or (false)
                                use the first valid frame as the zero angle.
                                (default: true)
    feathering:                 Set the amount of feathering for the edge
                                pixels of the bodypart image.  Setting 
                                feathering=0.0 gives a hard edge, and 
                                feathering=1.0 gives a very smooth edge.
                                This parameter affects motion tracking when
                                using area tracking, and small bodyparts may
                                benefit from setting feathering=0.0.
                                (default: 0.25)

Parameters that apply to Edge & Tip tracking:
    threshold:                  This is the threshold for the intensity
                                gradient to detect a bodypart edge or tip.
                                Typical values will be either 0.0 or something
                                small such as 0.01.  You may run the program 
                                in plot_tracker.py for help setting these 
                                thresholds, see the Plot Tracker section of
                                this document for details.
                                (default: 0.0)

Parameters that apply only to aux:
    wingbeat_min:               The lower wingbeat frequency (hz) of 
                                your insect.  (default: 180.0)
    wingbeat_max:               The upper wingbeat frequency (hz) of 
                                your insect.  (default: 220.0)

                             
Voltage Output
---------------------------
Kinefly can output voltages using a PhidgetsAnalog device using the ROS node flystate2phidgetsanalog (which should be launched under the ROS namespace of kinefly, i.e. <node name="flystate2phidgetsanalog" ... ns="kinefly" />).  The default values set the four voltage channels to (L, R, L-R, L+R), with the voltage value equal to the angle in radians.

PhidgetsAnalog parameters are specified in the 'params_kinefly.launch' file.  The parameters determine how the four voltage output channels set their voltages.  Each of the four channels is determined by a set of v#__ coefficients, where # is one of 0|1|2|3, for example, v0l1 is the leftangle1 component for voltage channel 0.

    v#enable:   Enable/disable each channel separately. (true|false)
    autorange:  Automatically set the four channels to L, R, L-R, L+R such
                that the voltages cover the range [-10,+10] based on the 
                min/max witnessed angles.


Each voltage channel # calculates a voltage by a linear combination of terms of the form:
 v#out =  v#l1 * left.angles[0]
        + v#l2 * left.angles[1]
        + v#r1 * right.angles[0]
        + v#r2 * right.angles[1]
        + v#ha * head.angles[0]
        + v#hr * head.radii[0]
        + v#aa * abdomen.angles[0]
        + v#ar * abdomen.radii[0]
        + v#xi * aux.intensity
       
Thus for example, a voltage output such as L-R can be generated on v2 by setting the v2l1=+1 and v2r1=-1.



Record Video:
---------------------------
During a Kinefly session, you may record a video to a .bag file, for later analysis, with a timestamped name such as '~/bagfiles/_2014-06-16-14-53-28.bag':
$ roslaunch Kinefly record.launch

You may also attach a prefix to the filename, such as '~/bagfiles/flynumberone_2014-06-16-14-53-28.bag':
$ roslaunch Kinefly record.launch prefix:=flynumberone



Retrack Prerecorded Video:
---------------------------
If environment var RETRACK is set to 1, and BAGFILE is set to a file (i.e. "export RETRACK=1" and "export BAGFILE=/path/to/file.bag"), then it uses the given .bag file instead of the camera(s).  An example, to retrack a single .bag file:                                           
$ export RETRACK=1                                      
$ export BAGFILE=/home/rancher/bagfiles/12345.bag       
$ roslaunch Kinefly main.launch                         

To retrack a bunch of bag files, use the Kinefly/retracker program.  An example, to retrack multiple .bag files:                                           
$ export BAGFILESPEC=/home/rancher/bagfiles/*.bag       
$ rosrun Kinefly retracker



LED Panels Control
---------------------------
Kinefly can control the MReiser LED Panels (see bitbucket.org/mreiser/panels, and github.com/ssafarik/ledpanels) either directly over USB, or indirectly via the voltage signal generated by the voltage output above.

In either case the ROS node from github.com/ssafarik/ledpanels must be running in order to send serial commands to the panels.  If you want to control the panels via USB, then you'll also need to run the ROS node flystate2ledpanels (which should be launched under the ROS namespace of kinefly, i.e. <node name="flystate2ledpanels ... ns="kinefly" />).
 
LEDPanels parameters are specified in the 'params_kinefly.launch'.  The parameters determine how the four voltage ADC input channels interpret the voltages into position and velocity commands. xpos, xvel, ypos, and yvel are determined by a set of adc# coefficients, where # is one of 0|1|2|3.

Each x or y channel calculates a position or velocity by a linear combination of terms of the form:
    Q = adc0*bnc0 
      + adc1*bnc1 
      + adc2*bnc2 
      + adc3*bnc3 
      + funcx*fx(t) 
      + funcy*fy(t),
where fx(t) and fy(t) are the internal panel controller functions that default to constant values of 10, unless otherwise changed.

The value Q then applies to either x or y as specified by the 'axis' parameter, and is interpreted as a position or velocity as specified by the 'mode' parameter.
 

ROS Commands
---------------------------
The kinefly/command ROS topic accepts the following string commands:
  help                 Shows this message.
  gui_on               Turn on the graphical user interface.
  gui_off              Turn off the graphical user interface.
  exit                 Exit the program.

You can send the above commands at the shell prompt, for example:
rostopic pub -1 kinefly/command Kinefly/MsgCommand commandtext help
rostopic pub -1 kinefly/command Kinefly/MsgCommand commandtext gui_on
rostopic pub -1 kinefly/command Kinefly/MsgCommand commandtext gui_off
rostopic pub -1 kinefly/command Kinefly/MsgCommand commandtext exit


Plot Tracker (auxillary program to view tracking performance)
---------------------------
The program plot_tracker.py will open a plot window that may help in analysing the performance of the kinematics tracking.  The plot shows a live view of internal tracker data, such as intensity profiles and thresholds, and this information may help you in setting various tracker parameters, such as threshold.

Each Kinefly tracker (edge, tip, etc) makes its data available via a service name such as 'trackerdata_head', 'trackerdata_left', 'trackerdata_right', or 'trackerdata_abdomen'.  

Prior to running plot_tracker, set the parameter 'trackername' to the signal of interest.  For example:
$ rosparam set trackername trackerdata_left
$ rosrun Kinefly plot_tracker.py