forked from ptweir/Kinefly
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathREADME
377 lines (286 loc) · 21.9 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
Kinefly
----------------
Kinefly extracts kinematics variables from the camera image of a tethered winged insect.
You may track the movement of each bodypart: head, abdomen, leftwing, rightwing. Several tracking techniques are available to choose from (area tracking, edge finding, tip tracking). An auxillary elliptical region may also be monitored (intended to capture leg motion or wingbeat frequency).
The software runs on ROS, the Robot Operating System, and can use any camera or other source that provides a ROS image stream, e.g. /camera/image_raw.
Contents:
---------------------------
Installation
Running
Main Window
Handle Positioning
Output Variables
Units
Bodypart Trackers
- Area Tracking
- Edge Tracking
- Tip Tracking
- Intensity Tracking and Wingbeat Frequency
Frame Rate
White on Black
Background Subtraction
Parameters
Voltage Output
Record Video for Later
Retrack Prerecorded Video
LED Panels
ROS Commands
Plot Tracker (auxillary program to view tracking performance)
Installation:
---------------------------
You need:
* ROS
* camera to supply a ROS image stream
* Kinefly (at github.com/ssafarik/Kinefly)
* ledpanels (at github.com/ssafarik/ledpanels)
* phidgets (at github.com/ssafarik/phidgets)
The Kinefly code is independent of the specific hardware and settings that you're using. This is accomplished by using parameters that are specific to your hardware (i.e. RIG). The parameters are set in a subdirectory of the Kinefly/launch directory, and each machine sets an environment variable 'RIG' that tells Kinefly which set of parameters to use.
If you want to output voltages (e.g. voltages to indicate wing angles, etc), then you'll need the PhidgetsAnalog hardware w/ Phidgets API. If you want to have the Kinefly measurements control LED panels, then you'll need panels hardware, see below. There might be more stuff, but that's the basic setup.
Refer to the file 'install.txt' for detailed instructions.
Running:
---------------------------
To run Kinefly, you first need to ensure that roscore is running. In a terminal window (note below that the $ is the command prompt, not typed by you):
$ roscore
Next, depending on if you have an ethernet or firewire or USB camera, edit the Kinefly/launch/RIG/source_camera.launch file as appropriate (where 'RIG' is the name of your rig). You may also need to edit the 'params_camera.launch' or 'params_kinefly.launch' files to set exposure, framerate, tracking params, etc.
Then in another terminal window use the roslaunch command to launch Kinefly:
$ roslaunch Kinefly main.launch
Main Window
---------------------------
The main window provides an interface to most, but not all, of the functionality. The buttons and checkboxes on the toolbar, and the handles on the image, function as follows:
<Exit> Terminate the program. Use this instead of the
titlebar's [x].
<SaveBG> Save the instant camera image to disk as ~/kinefly.png
Track:
[]H Turn on/off head tracking.
[]A Turn on/off abdomen tracking.
[]L Turn on/off left wing tracking.
[]R Turn on/off right wing tracking.
[]X Turn on/off the 'aux' area intensity & wingbeat frequency
calculations.
Subtract:
[]H Turn on/off background subtraction on head
[]A Turn on/off background subtraction on abdomen
[]LR Turn on/off background subtraction on left & right wings
[]X Turn on/off background subtraction on aux
[]Stabilize Turn on/off image stabilization for head & abdomen. The
quality of stabilization is very dependent upon the region
selected with the handles. (default: false)
[]Symmetric When moving handles, keep them symmetric about the body axis.
[]Windows Turn on/off helpful windows showing various internal images.
Handle Positioning:
---------------------------
Each of the head/abdomen/left/right body parts has a set of control points that determine what part of the image is tracked. The handle positions determine the hinges, angles, and radii.
First, for best tracking, each of the head/abdomen/left/right body parts should have the 'hinge' handle placed at the center of rotation, which for example in the case of the abdomen is not the center of the abdomen, but rather is the point where the abdomen connects to the rest of the body, and rotates about.
For the head and abdomen, set the inner and outer radii to cover just a narrow strip that encloses unique features of the bodypart as the bodypart rotates, and set the angles to cover all of the bodypart motion. Reliable head rotation seems to be the most dependent on proper handle positions.
For the wings, the angles and radii should be positioned to indicate a region of interest such that legs and other non-wing moving things do not encroach into the region of interest.
Output Variables
---------------------------
The output is published on the ROS topic '/kinefly/flystate', and can be retrieved either by running the command 'rostopic echo kinefly/flystate' or by writing your own ROS node and subscribing to that topic.
The fields shown in [brackets] are lists of values, and there may be zero, one, or more than one entry (e.g. when finding multiple wing edges). Which fields contain valid values are dependent upon which bodypart tracker has been set, and its settings.
An example of the flystate output is shown here:
---
header:
seq: 1033
stamp:
secs: 1393885525
nsecs: 419325113
frame_id: Fly
head:
angles: [-0.05834060178390743]
gradients: []
radii: [54.635968488535795]
freq: 0.0
intensity: 0.45413453997
abdomen:
angles: [-0.37292074013475407]
gradients: []
radii: [116.38383782940254]
freq: 0.0
intensity: 0.591699666712
left:
angles: [0.8233633148061248]
gradients: [-0.00784313678741455]
radii: []
freq: 0.0
intensity: 0.802197675672
right:
angles: [0.6048098145857046]
gradients: [0.011551618576049805]
radii: []
freq: 0.0
intensity: 0.593512067496
aux:
angles: []
gradients: []
radii: []
freq: 0.0
intensity: 0.673986792564
Units
---------------------------
Angle units are radians.
Radius units are pixels.
Intensity is on the range [0,1], where 0.0 equals all pixels black, and 1.0 equals all pixels white. The main window shows intensity units in parentheses, ().
Gradient is the intensity change per unit, and is used for edge and tip finding. Edge-tracking gradient is change per angle, and tip-tracking gradient is change per radial distance.
Each bodypart (head, abdomen, and wings) has its own angle origin, which in general terms is outward from the body center. The zero angle for the head is directly opposite from the abdomen, and the zero of the abdomen angle is directly opposite from the head. The zero for each wing is directly opposite from the other wing hinge. In other words, imagine a line connecting the head and abdomen hinges, and a line connecting each wing hinge. The zero angles are outward along those lines, and the range of each angle is on the range [-pi,+pi].
Bodypart Trackers:
---------------------------
The trackers available are as follows, and are specified via the 'tracker' parameter in the params_kinefly.launch file. Any bodypart (except aux) can use any tracker.
'area' Uses image registration techniques, and reports the rotation and
radial movement of the area as a whole.
'edge' Finds radial edges, ordered by decreasing edge strength.
'tip' Finds the point where the bodypart is farthest from
the rotation center.
'intensity' Monitor the pixel intensity, and the frequency of periodic
intensity, of an elliptical area
There is an auxillary Plot Tracker tool that may be helpful for setting wing edge detection thresholds, etc. See the Plot Tracker section of this document for details.
Area Tracking:
---------------------------
This is the default for the head & abdomen. We measure rotation and radial expansion of the bodypart via an image registration technique in polar coordinates on the region of interest. This state measurement is autozero'ed such that the origin tends to be near the midpoint of the range of motion. Autoranging will therefore cause occasional changes of the angle origin, making it seem as though there were a step change in the angle when there wasn't. Autoranging can be turned off by changing the parameter setting.
Area tracking is also affected by a "feathering" (i.e. a window function) parameter that smooths the edge pixels of the bodypart. Setting feathering=0 gives a rectangular window (i.e. a hard edge), and feathering=1 gives a Hanning window (i.e. a very smooth edge). Tracking of small body parts, such as a fly head, may benefit from setting feathering=0. The default is 0.25.
Edge Tracking:
---------------------------
This is the default for left & right wings. We find multiple edge angles per wing (you can use this feature to track legs etc too, if desired), the number of edges being specified via parameter 'n_edges_max'. The edge angles are in one list, and their corresponding intensity gradients are in another. The edges are found by looking at the intensity gradient across the wingstroke, and each list is sorted in descending order of absolute gradient.
The parameter 'threshold' can be used to reduce false detections when no edge is present, and will typically be a low value such as 0.01. It is set independently for each wing.
Tip Tracking:
---------------------------
When the 'tip' tracker is set for a bodypart, We locate the bodypart point farthest from the point of rotation. The parameter 'threshold' is compared to the radial gradient to find the bodypart, and should be set to approx 0.01.
Intensity Tracking and Wingbeat Frequency:
---------------------------
We make an attempt to measure the wingbeat frequency using the frequency spectrum of the aux intensity. For this to work, several criteria must be met. First, you must set the valid range of wingbeat frequencies in the params_kinefly.launch file. Upon the next launch, Kinefly will print out the compatible camera framerate needed (the lowest typically being slightly higher than twice the wingbeat passband width). Next, set the camera to that framerate. If the camera framerate can be maintained within the valid range, then Kinefly will compute the wingbeat frequency. Since the framerate is typically much less than the Nyquist frequency needed to cover the highest wingbeat frequency, we use an undersampling technique to sample just the passband, and look at a baseband alias of the passband frequencies. Proper camera framerate is critical to the functioning of this feature, and the computer must be fast enough to process the frames in the allowed time, with no camera framerate variation outside of the allowed range. If you have trouble maintaining the framerate, consider using the parameter 'scale_image'.
Frame Rate
---------------------------
The frame rate is dependent upon virtually every setting. Turn off tracking of unneeded bodyparts, use the smallest usable regions of interest (i.e. handle positions), and turn off unneeded windows to speed the frame rate. The largest improvement is likely to come from lowering the value of 'scale_image' in the params_kinefly.launch file.
White On Black
---------------------------
Kinefly works best internally if the image is white-on-black. The camera image may be either black-on-white or white-on-black, and whether to invert the color or not is decided automatically based on the pixel intensity of a region centered on the intersection of the lines head/abdomen hinges and left/right hinges. If the mean pixel value of this region is darker than the mean pixel value of the overall image, then we invert the colors.
The user can override the autodetect and set the behavior manually via the checkbox on the toolbar. When Kinefly launches it starts in autodetect mode, but once the checkbox is clicked, then the manual setting is used until the next launch of Kinefly. The setting is not saved across launches.
Background Subtraction
---------------------------
Background subtraction can be turned on/off for each bodypart individually. The background is initialized with the kinefly.png image on disk, and updates as a moving average with the time constant (in seconds) parameter rc_background.
Saturation of the background image can cause problems with tracking, due to limited foreground headroom. Kinefly contains a "saturation correction" algorithm to reduce or eliminate this effect, however the feature is turned off by default due to the computational cost and reduced framerate. You may turn it on/off for each bodypart separately via the parameter "saturation_correction". The feature works by reducing the importance of pixels the closer they are to saturated.
Parameters.
---------------------------
Parameters are stored in the file 'Kinefly/launch/params_kinefly.launch'.
Anything set via the GUI (e.g. handle positions, checkbox states, etc) are automatically saved to the kinefly.yaml file.
The non-gui parameters are below.
General Parameters:
filenameBackground: Location to store the background image.
(default: ~/kinefly.png)
image_topic: ROS topic for the image stream. Running
the ROS node image_proc in addition to
Kinefly and setting this to image_mono
may help with some cameras that provide
bayer encoded images.
(default: /camera/image_raw)
n_queue_images: Length of the circular buffer for incoming
images. (default: 2)
n_edges_max: Max number of wing edges to find. (default: 1)
rc_background: Time constant of the moving average background
image. (default: 1000.0)
scale_image: Scale all images by this amount. Smaller
values give better framerate. (default: 1.0)
use_gui: Turn on/off the graphical user interface to
speed the framerate. (default: true)
parameterfile: Location of the .yaml file for keeping track
of GUI handle positions & checkboxes.
(default: ~/kinefly.yaml)
Head/Abdomen/Left/Right Parameters:
tracker: Which feature to track. Options are 'area',
'edge', and 'tip'.
(default: area/area/edge/edge)
saturation_correction: Correct for bright pixels in the background
image to improve tracking. (default: false)
Parameters that apply to Area tracking:
autozero: Automatically (true) determine the zero
angle based on center of motion, or (false)
use the first valid frame as the zero angle.
(default: true)
feathering: Set the amount of feathering for the edge
pixels of the bodypart image. Setting
feathering=0.0 gives a hard edge, and
feathering=1.0 gives a very smooth edge.
This parameter affects motion tracking when
using area tracking, and small bodyparts may
benefit from setting feathering=0.0.
(default: 0.25)
Parameters that apply to Edge & Tip tracking:
threshold: This is the threshold for the intensity
gradient to detect a bodypart edge or tip.
Typical values will be either 0.0 or something
small such as 0.01. You may run the program
in plot_tracker.py for help setting these
thresholds, see the Plot Tracker section of
this document for details.
(default: 0.0)
Parameters that apply only to aux:
wingbeat_min: The lower wingbeat frequency (hz) of
your insect. (default: 180.0)
wingbeat_max: The upper wingbeat frequency (hz) of
your insect. (default: 220.0)
Voltage Output
---------------------------
Kinefly can output voltages using a PhidgetsAnalog device using the ROS node flystate2phidgetsanalog (which should be launched under the ROS namespace of kinefly, i.e. <node name="flystate2phidgetsanalog" ... ns="kinefly" />). The default values set the four voltage channels to (L, R, L-R, L+R), with the voltage value equal to the angle in radians.
PhidgetsAnalog parameters are specified in the 'params_kinefly.launch' file. The parameters determine how the four voltage output channels set their voltages. Each of the four channels is determined by a set of v#__ coefficients, where # is one of 0|1|2|3, for example, v0l1 is the leftangle1 component for voltage channel 0.
v#enable: Enable/disable each channel separately. (true|false)
autorange: Automatically set the four channels to L, R, L-R, L+R such
that the voltages cover the range [-10,+10] based on the
min/max witnessed angles.
Each voltage channel # calculates a voltage by a linear combination of terms of the form:
v#out = v#l1 * left.angles[0]
+ v#l2 * left.angles[1]
+ v#r1 * right.angles[0]
+ v#r2 * right.angles[1]
+ v#ha * head.angles[0]
+ v#hr * head.radii[0]
+ v#aa * abdomen.angles[0]
+ v#ar * abdomen.radii[0]
+ v#xi * aux.intensity
Thus for example, a voltage output such as L-R can be generated on v2 by setting the v2l1=+1 and v2r1=-1.
Record Video:
---------------------------
During a Kinefly session, you may record a video to a .bag file, for later analysis, with a timestamped name such as '~/bagfiles/_2014-06-16-14-53-28.bag':
$ roslaunch Kinefly record.launch
You may also attach a prefix to the filename, such as '~/bagfiles/flynumberone_2014-06-16-14-53-28.bag':
$ roslaunch Kinefly record.launch prefix:=flynumberone
Retrack Prerecorded Video:
---------------------------
If environment var RETRACK is set to 1, and BAGFILE is set to a file (i.e. "export RETRACK=1" and "export BAGFILE=/path/to/file.bag"), then it uses the given .bag file instead of the camera(s). An example, to retrack a single .bag file:
$ export RETRACK=1
$ export BAGFILE=/home/rancher/bagfiles/12345.bag
$ roslaunch Kinefly main.launch
To retrack a bunch of bag files, use the Kinefly/retracker program. An example, to retrack multiple .bag files:
$ export BAGFILESPEC=/home/rancher/bagfiles/*.bag
$ rosrun Kinefly retracker
LED Panels Control
---------------------------
Kinefly can control the MReiser LED Panels (see bitbucket.org/mreiser/panels, and github.com/ssafarik/ledpanels) either directly over USB, or indirectly via the voltage signal generated by the voltage output above.
In either case the ROS node from github.com/ssafarik/ledpanels must be running in order to send serial commands to the panels. If you want to control the panels via USB, then you'll also need to run the ROS node flystate2ledpanels (which should be launched under the ROS namespace of kinefly, i.e. <node name="flystate2ledpanels ... ns="kinefly" />).
LEDPanels parameters are specified in the 'params_kinefly.launch'. The parameters determine how the four voltage ADC input channels interpret the voltages into position and velocity commands. xpos, xvel, ypos, and yvel are determined by a set of adc# coefficients, where # is one of 0|1|2|3.
Each x or y channel calculates a position or velocity by a linear combination of terms of the form:
Q = adc0*bnc0
+ adc1*bnc1
+ adc2*bnc2
+ adc3*bnc3
+ funcx*fx(t)
+ funcy*fy(t),
where fx(t) and fy(t) are the internal panel controller functions that default to constant values of 10, unless otherwise changed.
The value Q then applies to either x or y as specified by the 'axis' parameter, and is interpreted as a position or velocity as specified by the 'mode' parameter.
ROS Commands
---------------------------
The kinefly/command ROS topic accepts the following string commands:
help Shows this message.
gui_on Turn on the graphical user interface.
gui_off Turn off the graphical user interface.
exit Exit the program.
You can send the above commands at the shell prompt, for example:
rostopic pub -1 kinefly/command Kinefly/MsgCommand commandtext help
rostopic pub -1 kinefly/command Kinefly/MsgCommand commandtext gui_on
rostopic pub -1 kinefly/command Kinefly/MsgCommand commandtext gui_off
rostopic pub -1 kinefly/command Kinefly/MsgCommand commandtext exit
Plot Tracker (auxillary program to view tracking performance)
---------------------------
The program plot_tracker.py will open a plot window that may help in analysing the performance of the kinematics tracking. The plot shows a live view of internal tracker data, such as intensity profiles and thresholds, and this information may help you in setting various tracker parameters, such as threshold.
Each Kinefly tracker (edge, tip, etc) makes its data available via a service name such as 'trackerdata_head', 'trackerdata_left', 'trackerdata_right', or 'trackerdata_abdomen'.
Prior to running plot_tracker, set the parameter 'trackername' to the signal of interest. For example:
$ rosparam set trackername trackerdata_left
$ rosrun Kinefly plot_tracker.py