-
Notifications
You must be signed in to change notification settings - Fork 5
ErrorMeas
This chapter explains how to verify recorded data in terms of network performance.
- The caffe framework from this repository is build and installed correctly.
- Recorded data are available (for example the training data from the original DeepDriving project). See: here
For an easy start, a couple of environment variables can be defined:
-
DEEPDRIVING_CAFFE_PATH
must contain the path to the caffe-installation.
export DEEPDRIVING_CAFFE_PATH=<path-to-installation>
-
(optional)
DEEPDRIVING_DATABASE_PATH
must contain a path to the recorded data (for example the trainings data). If this variable is not defined, the database must be provided as first argument of the verification script. -
(optional)
DEEPDRIVING_MODEL_PATH
must contain a path to the caffe model (for example<repository-path>/torcs/pre_trained/driving_run_1F.prototxt
). If this variable is not defined, the pre-trained model from this repository is used. -
(optional)
DEEPDRIVING_WEIGHTS_PATH
must contain a path to the trained caffe weights of the model (for example<repository-path>/torcs/pre_trained/driving_train_1F_iter_140000.caffemodel
). If this variable is not defined, the pre-trained model from this repository is used. -
(optional)
DEEPDRIVING_MEAN_PATH
must contain a path to the mean-file which is used to normalise the input images for the network (for example<repository-path>/torcs/pre_trained/driving_mean_1F.binaryproto
). If this variable is not defined, the mean-file from this repository is used (which has been generated out of the DeepDriving training data). -
(optional)
DEEPDRIVING_GPU
must contain the ID of the GPU which should be used for inference. On a single GPU system, this ID is normally 0. If this variable is not defined or contains the value -1, the CPU is used instead of the GPU, which leads to bad performance. Note, that Caffe must be compiled for GPU support with CUDA and cuDNN to allow the DeepDriving network calculating on GPU.
- Run the verification script from the caffe-repository:
-
Note: The first parameter is only necessary if another database is used than the database defined by
DEEPDRIVING_DATABASE_PATH
.
-
Note: The first parameter is only necessary if another database is used than the database defined by
cd <repository-path>/torcs
./torcs_verify.sh <path-to-database>
- You should see a window which shows a semantic situation description (lanes, position of host car, distance to traffic cars) and the corresponding frame from the database. On the terminal you can see the label-values for every frame.
-
At the start the application is in PAUSE mode. Focus the window and press
p
to disable the PAUSE mode. -
The following shortcuts are possible:
-
p
: Enable or disable PAUSE mode. -
n
: Show next frame. -
+
: If the PAUSE mode is disabled, it increases the frame speed. -
-
: If the PAUSE mode is disabled, it decreases the frame speed. -
ESC
: Close application.
-
-
The verification tool exists automatically, if all data from the database have been processed.
-
It also exists by pressing
ESC
. -
In both cases the application prints the result of the error measurement to the terminal:
Error Measurements:
Type MAE SD MSE
DistLL 0.0262877 0.0469189 0.00289243
DistMM 0.0191372 0.0304409 0.00129288
DistRR 0.0270002 0.0409293 0.00240422
LL 0.0177536 0.024211 0.000901364
ML 0.0172968 0.0288982 0.00113429
MR 0.0177384 0.037068 0.00168869
RR 0.0212317 0.0331885 0.00155226
DistL 0.0184354 0.0324393 0.00139217
DistR 0.018433 0.0324366 0.00139191
L 0.0231213 0.0480441 0.00284283
M 0.0240487 0.0463001 0.00272204
R 0.0226622 0.0503326 0.00304695
Angle 0.0195737 0.018004 0.000707273
Fast 0.00339844 0.0450279 0.00203906
Sum 0.138059 0.25712 0.0130042
Number of Error Measurements: 5120
-
There is also a fast verification tool available, which does not process each frame individually by the network, but uses a batch-size of 64 frames. This tool does not have a graphical output. It outputs the current mean error measurements after every 1000 frames.
-
Run the verification script from the caffe-repository:
-
Note: The first parameter is only necessary if another database is used then the database defined by
DEEPDRIVING_DATABASE_PATH
.
-
Note: The first parameter is only necessary if another database is used then the database defined by
cd <repository-path>/torcs
./torcs_verify_fast.sh <path-to-database>
- The tool exists automatically when all frames were processed. Alternatively it can be closed by pressing CTRL+C.
-
There are three types of error measures calculated: The mean absolute error (MAE) and the standard deviation (SD) of the absolute error. These are the error types used by the author of the DeepDriving project to compare different models with each other.
-
Furthermore also the mean squared error is calculated. The sum of the mean squared error of all indicators is used as loss for optimizing the network during training.
-
Here are some example error measurements (Loss):
Checkpoint | Training-Set Loss | Test-Set Loss (TORCS_Caltech_1F_Testing_280) |
---|---|---|
10000 Iterations | 0.0479343 | 0.133739 |
140000 Iterations (pre-trained) | 0.0191592 | 0.0978131 |
170000 Iterations | 0.0177159 | 0.1063 |