Tutorial 2 Launch and train CDeep3M

This page provides instructions on how to train CDeep3M using training data generated in:

Tutorial 1: Generating training data with IMOD

Goals

1. Launch CDeep3M instance in Amazon Web Services (AWS)
2. Upload training data to CDeep3M instance
3. Run CDeep3M train
4. Check on CDeep3M train status

Requirements

• Command line terminal (on Windows the Cygwin Terminal that is installed with IMOD will work just fine)
• ssh client (on Windows the Cygwin Terminal that is installed with IMOD will work just fine)
• Amazon Web Services (AWS) account

Step 1 Launch CDeep3M instance on AWS

WARNING: The instructions on link below will launch a virtual machine on AWS and charges will be incurred.

Follow these instructions to launch CDeep3M instance on AWS

Step 2 Download dataset two

From the previous tutorial you should have generated training data which is stored in the train directory and you should have a terminal open where the train directory is visible as seen here:

If not, click here to download training data that can be used for this tutorial.

Step 3 Upload train directory to CDeep3M instance

A. Upload train directory to CDeep3M instance created in Step 1 <PublicDNS> can be found circled in red on Step 9 from Launching CDeep3M via AWS CloudFormation

ls train
scp -i ~/.ssh/id_rsa -r train ubuntu@<PublicDNS value from AWS>:/home/ubuntu/.

B. Connect to CDeep3M instance

ssh -i ~/.ssh/id_rsa ubuntu@<PublicDNS value from AWS>

C. Verify train directory was uploaded

ls train

Step 4 Preprocess training data

A. Preprocess training data by running PreprocessTrainingData.m command which takes three arguments in order; images directory, labels directory, and output directory:

PreprocessTrainingData.m ~/train/images ~/train/labels ~/augtrain

Click here for more information about PreprocessTrainingData.m

Output:

octave: X11 DISPLAY environment variable not set
octave: disabling GUI features
Starting Training data Preprocessing
Training Image Path:
/home/ubuntu/train/images
Training Label Path:
/home/ubuntu/train/labels/
Output Path:
/home/ubuntu/augtrain
Loading:
/home/ubuntu/train/images
Image importer loading ...
/home/ubuntu/train/images
Reading file: /home/ubuntu/train/images/x.000.png
.
.
Verifying labels
Checking image dimensions
Augmenting training data 1-8 and 9-16

Create variation 1 and 9
Saving: /home/ubuntu/augtrain/training_full_stacks_v1.h5
Saving: /home/ubuntu/augtrain/training_full_stacks_v9.h5
.
.
Create variation 8 and 16
Saving: /home/ubuntu/augtrain/training_full_stacks_v8.h5
Saving: /home/ubuntu/augtrain/training_full_stacks_v16.h5
Elapsed time is 9.76223 seconds.
-> Training data augmentation completed
Training data stored in /home/ubuntu/augtrain
For training your model please run runtraining.sh /home/ubuntu/augtrain <desired output directory>

Step 5 Run CDeep3M train job

A. Since training can take a while (hours to even days) we will be using the screen command so we can disconnect. Information about screen can be found here or by typing man screen from the terminal. Type screen as seen below.

screen

Hit space bar key to continue

B. CDeep3M training is done by invoking runtraining.sh command. In the example below we will be training for 20 iterations which is too short to be of any value, but shows the process. Type the following command:

runtraining.sh --numiterations 20 ~/augtrain ~/model

Click here for more information about runtraining.sh

Output:

octave: X11 DISPLAY environment variable not set
octave: disabling GUI features
Verifying input training data is valid ... success
Copying over model files and creating run scripts ... success

A new directory has been created: /home/ubuntu/model
In this directory are 3 directories 1fm,3fm,5fm which
correspond to 3 caffe models that need to be trained

Single GPU detected.

Training has completed. Have a nice day!


Training has completed. Results are stored in /home/ubuntu/model
Have a nice day!

C. Detach screen from CDeep3M training job. This can be done by hitting the key combination Control-a then the d key or in shorter form: Ctrl-a d

Step 6 Check CDeep3M train job status

A. Re-attach to screen running CDeep3M training job by typing screen -r as seen here:

screen -r

Once training has finished go to the next step.

Step 7 Optionally download trained model

A. Exit from CDeep3M instance by typing exit. This might need to be done twice since we are in a screen

exit