Tutorial 2 Launch and train CDeep3M

This page provides instructions on how to perform augmented training of CDeep3M using training data generated in:

Tutorial 1: Generating training data with IMOD

Goals

1. Launch CDeep3M instance in Amazon Web Services (AWS)
2. Upload training data to CDeep3M instance
3. Run CDeep3M train to perform augmented training
4. Check on CDeep3M train status

Requirements

• Command line terminal (on Windows the Cygwin Terminal that is installed with IMOD will work)
• ssh client (on Windows the Cygwin Terminal that is installed with IMOD will work)
• Amazon Web Services (AWS) account

Step 1 Launch CDeep3M instance on AWS

WARNING: The instructions on link below will launch a virtual machine on AWS and charges will be incurred.

Follow these instructions to launch CDeep3M instance on AWS

Step 2 Download dataset two

From the previous tutorial you should have generated training data which is stored in the train directory and you should have a terminal open where the train directory is visible as seen here:

If not, click here to download training data that can be used for this tutorial.

Step 3 Upload train directory to CDeep3M instance

A. Upload train directory to CDeep3M instance created in Step 1 <PublicDNS> can be found circled in red on Step 9 from Launching CDeep3M via AWS CloudFormation

ls train
scp -i ~/.ssh/id_rsa -r train ubuntu@<PublicDNS value from AWS>:/home/ubuntu/.

B. Connect to CDeep3M instance

ssh -i ~/.ssh/id_rsa ubuntu@<PublicDNS value from AWS>

C. Verify train directory was uploaded

ls train

Step 4 Preprocess training data

A. Preprocess training data by running PreprocessTrainingData.m command which takes three arguments in order; images directory, labels directory, and output directory:

PreprocessTrainingData.m ~/train/images ~/train/labels ~/augtrain

Click here for more information about PreprocessTrainingData.m

Output:

octave: X11 DISPLAY environment variable not set
octave: disabling GUI features
Starting Training data Preprocessing
Training Image Path:
/home/ubuntu/train/images
Training Label Path:
/home/ubuntu/train/labels/
Output Path:
/home/ubuntu/augtrain
Loading:
/home/ubuntu/train/images
Image importer loading ...
/home/ubuntu/train/images
Reading file: /home/ubuntu/train/images/x.000.png
.
.
Verifying labels
Checking image dimensions
Augmenting training data 1-8 and 9-16

Create variation 1 and 9
Saving: /home/ubuntu/augtrain/training_full_stacks_v1.h5
Saving: /home/ubuntu/augtrain/training_full_stacks_v9.h5
.
.
Create variation 8 and 16
Saving: /home/ubuntu/augtrain/training_full_stacks_v8.h5
Saving: /home/ubuntu/augtrain/training_full_stacks_v16.h5
Elapsed time is 9.76223 seconds.
-> Training data augmentation completed
Training data stored in /home/ubuntu/augtrain
For training your model please run runtraining.sh /home/ubuntu/augtrain <desired output directory>

Step 5 Run CDeep3M train job

A. Since training can take a while (hours to even days) we will be using the screen command so we can disconnect. Information about screen can be found here or by typing man screen from the terminal. Type screen as seen below.

screen

Hit enter/return key to continue

B. CDeep3M training is done by invoking runtraining.sh command. In the example below we will be training for 20 iterations which is too short to be of any value, but shows the process. Type the following command:

runtraining.sh --additerations 100 --retrain ~/sbem/mitochrondria/xy5.9nm40nmz/30000iterations_train_out  ~/augtrain ~/model

Click here for more information about runtraining.sh

Output:

octave: X11 DISPLAY environment variable not set
octave: disabling GUI features
Verifying input training data is valid ... success
Copying over model files and creating run scripts ... success

A new directory has been created: /home/ubuntu/model
In this directory are 3 directories 1fm,3fm,5fm which
correspond to 3 caffe models that need to be trained

Single GPU detected.

Training has completed. Have a nice day!


Training has completed. Results are stored in /home/ubuntu/model
Have a nice day!

C. Detach screen from CDeep3M training job. This can be done by hitting the key combination Control-a then the d key or in shorter form: Ctrl-a d

D. Re-attach to screen running CDeep3M training job by typing screen -r as seen here:

screen -r

Once training has finished go to the next step. A completed train job will have output text as seen above.

Step 6 Optionally download trained model

A. Exit from CDeep3M instance by typing exit. exit will need to be done twice since we are in a screen virtual terminal.

exit
exit

B. Download the trained model by using the scp command as seen here:

scp -i ~/.ssh/id_rsa -r ubuntu@<PublicDNS value from AWS>:/home/ubuntu/model .
ls

Next

Congratulations on completing Tutorial 2

Click here to continue with Tutorial 3: Run CDeep3M

NOTE: If you are not continuing to the next tutorial be sure to shutdown your CDeep3M to avoid incurring additional EC2 charges.

Instructions for shutdown can be found here