ErrorMeas

Verify Recorded Data

This chapter explains how to verify recorded data in terms of network performance.

Pre-Requirements

• The caffe framework from this repository is build and installed correctly.
• Recorded data (for example the trainings data from the original DeepDriving project). See: here

Environment Setup

For an easy start, a couple of environment variables can be defined:

• DEEPDRIVING_CAFFE_PATH must contain the path to the caffe-installation.

export DEEPDRIVING_CAFFE_PATH=<path-to-installation>

• (optional) DEEPDRIVING_DATABASE_PATH must contain a path to the recorded data (for example the trainings data). If this variable is not defined, the database must be provided as first argument of the verification script.

• (optional) DEEPDRIVING_MODEL_PATH must contain a path to the caffe model (for example <repository-path>/torcs/pre_trained/driving_run_1F.prototxt). If this variable is not defined, the pre-trained model from this repository is used.

• (optional) DEEPDRIVING_WEIGHTS_PATH must contain a path to the trained caffe weights of the model (for example <repository-path>/torcs/pre_trained/driving_train_1F_iter_140000.caffemodel). If this variable is not defined, the pre-trained model from this repository is used.

• (optional) DEEPDRIVING_MEAN_PATH must contain a path to the mean-file which is used to normalise the input images for the network (for example <repository-path>/torcs/pre_trained/driving_mean_1F.binaryproto). If this variable is not defined, the mean-file from this repository is used (which has been generated out of the DeepDriving training data).

• (optional) DEEPDRIVING_GPU must contain the ID of the GPU which should be used for inference. On a single GPU system, this ID is normally 0. If this variable is not defined or contains the value -1, the CPU is used instead of the GPU, which leads to a bad performance. Note, that Caffe must be compiled for GPU support with CUDA and cuDNN to allow the DeepDriving network to be calculated on GPU.

Start the (graphical) Database Verification Tool

• Run the verification script from the caffe-repository:
  • Note: The first parameter is only necessary if another database is used then the database defined by DEEPDRIVING_DATABASE_PATH.

cd <repository-path>/torcs
./torcs_verify.sh <path-to-database>

• You should see a window which shows a semantic situation description (lanes, position of host car, distance to traffic cars) and the corresponding frame from the database. On the terminal you can see the label-values for every frame.

• At the start the application is in PAUSE mode. Focus the window and press p to disable the PAUSE mode.

• The following shortcuts are possible:

  • p: Enable or disable PAUSE mode.
  • n: Show next frame.
  • +: If the PAUSE mode is disabled, it increases the frame speed.
  • -: If the PAUSE mode is disabled, it decreases the frame speed.
  • ESC: Close application.

• The verification tool exists automatically, if all data from the database have been processed.

• It also exists by pressing ESC.

• In both cases the application prints the result of the error measurement to the terminal:

Error Measurements:
 Type   	 MAE 		 SD 		 MSE 
 DistLL 	 0.0262877 	 0.0469189 	 0.00289243
 DistMM 	 0.0191372 	 0.0304409 	 0.00129288
 DistRR 	 0.0270002 	 0.0409293 	 0.00240422
 LL     	 0.0177536 	 0.024211 	 0.000901364
 ML     	 0.0172968 	 0.0288982 	 0.00113429
 MR     	 0.0177384 	 0.037068 	 0.00168869
 RR     	 0.0212317 	 0.0331885 	 0.00155226
 DistL  	 0.0184354 	 0.0324393 	 0.00139217
 DistR  	 0.018433 	 0.0324366 	 0.00139191
 L      	 0.0231213 	 0.0480441 	 0.00284283
 M      	 0.0240487 	 0.0463001 	 0.00272204
 R      	 0.0226622 	 0.0503326 	 0.00304695
 Angle  	 0.0195737 	 0.018004 	 0.000707273
 Fast   	 0.00339844 	 0.0450279 	 0.00203906
 Sum    	 0.138059 	 0.25712 	 0.0130042
Number of Error Measurements: 5120


*** Time Summary for the Neural Net ***
Number of Inferences: 693
Mean Process Time   : 0.0147964s
Mean Forward Time   : 0.01445s
Max Process Time   : 0.0399579s
Max Forward Time   : 0.0391422s

Start the (fast) Database Verification Tool

• There is also a fast verification tool available, which does not process each frame individually by the network, but uses a batch-size of 64 frames. This tool does not have a graphical output. It outputs the current mean error measurements after every 1000 frames.

• Run the verification script from the caffe-repository:

  • Note: The first parameter is only necessary if another database is used then the database defined by DEEPDRIVING_DATABASE_PATH.

cd <repository-path>/torcs
./torcs_verify_fast.sh <path-to-database>

• The tool exists aeronautically when all frames were processed. Alternatively it can be closed by pressing CTRL+C.

Error Measurement Examples

• There are three types of error measures calculated: The mean absolute error (MAE) and the standard deviation (SD) of the absolute error. These are the error types used by the author Mr Chen to compare different models with each other.

• Furthermore also the mean squared error is calculated. The sum of the mean squared error of all indicators is used as loss for optimising the network during training.

• Here are some example error measurements available (Loss):

Checkpoint	Training-Set Loss	Test-Set Loss (TORCS_Caltech_1F_Testing_280)
10000 Iterations	0.0479343	0.133739
140000 Iterations (pre-trained)	0.0191592	0.0978131
170000 Iterations	0.0177159	0.1063

Next Step

• Record new Data