Tutorial 2 Launch and train CDeep3M
This page provides instructions on how to perform augmented training of CDeep3M using training data generated in:
Tutorial 1: Generating training data with IMOD
- Launch CDeep3M instance in Amazon Web Services (AWS)
- Upload training data to CDeep3M instance
- Run CDeep3M train
- Check on CDeep3M train status
- Command line terminal (on Windows the Cygwin Terminal that is installed with IMOD will work)
- ssh client (on Windows the Cygwin Terminal that is installed with IMOD will work)
- Amazon Web Services (AWS) account
WARNING: The instructions on link below will launch a virtual machine on AWS and charges will be incurred.
A. Create keypair if one hasn't been made already.
Instructions about key pairs can be found here, but basically
Click here to login to AWS and import key pair in us-west-2 region
Follow these instructions to launch CDeep3M instance on AWS
From the previous tutorial you should have generated training data which is stored in the train directory and you should have a terminal open where the train directory is visible as seen here:
If not, click here to download training data that can be used for this tutorial.
A. Upload train directory to CDeep3M instance created in Step 1 <PublicDNS> can be found circled in red on Step 9 from Launching CDeep3M via AWS CloudFormation
ls train
scp -i ~/.ssh/id_rsa -r train ubuntu@<PublicDNS value from AWS>:/home/ubuntu/.
B. Connect to CDeep3M instance
ssh -i ~/.ssh/id_rsa ubuntu@<PublicDNS value from AWS>
C. Verify train directory was uploaded
ls train
A. Preprocess training data by running PreprocessTrainingData.m command which takes three arguments in order; images directory, labels directory, and output directory:
PreprocessTrainingData.m ~/train/images ~/train/labels ~/augtrainClick here for more information about PreprocessTrainingData.m
Output:
octave: X11 DISPLAY environment variable not set
octave: disabling GUI features
Starting Training data Preprocessing
Training Image Path:
/home/ubuntu/train/images
Training Label Path:
/home/ubuntu/train/labels/
Output Path:
/home/ubuntu/augtrain
Loading:
/home/ubuntu/train/images
Image importer loading ...
/home/ubuntu/train/images
Reading file: /home/ubuntu/train/images/x.000.png
.
.
Verifying labels
Checking image dimensions
Augmenting training data 1-8 and 9-16
Create variation 1 and 9
Saving: /home/ubuntu/augtrain/training_full_stacks_v1.h5
Saving: /home/ubuntu/augtrain/training_full_stacks_v9.h5
.
.
Create variation 8 and 16
Saving: /home/ubuntu/augtrain/training_full_stacks_v8.h5
Saving: /home/ubuntu/augtrain/training_full_stacks_v16.h5
Elapsed time is 9.76223 seconds.
-> Training data augmentation completed
Training data stored in /home/ubuntu/augtrain
For training your model please run runtraining.sh /home/ubuntu/augtrain <desired output directory>
A. Since training can take a while (hours to even days) we will be using the screen command so we can disconnect. Information about screen can be found here or by typing man screen from the terminal. Type screen as seen below.
screen
Hit enter/return key to continue
B. CDeep3M training is done by invoking runtraining.sh command. Since full training can take a couple days, the instructions below will be retraining the following pretrained model, already preloaded on the CDeep3M instance, ~/sbem/mitochrondria/xy5.9nm40nmz/30000iterations_train_out with training data we generated in tutorial 1. This is done by using the --retrain and --additerations flag as seen below. Type the following command:
runtraining.sh --additerations 20 --retrain ~/sbem/mitochrondria/xy5.9nm40nmz/30000iterations_train_out ~/augtrain ~/modelClick here for more information about runtraining.sh
Click here for more information about how to retrain a pretrained model
Output:
octave: X11 DISPLAY environment variable not set
octave: disabling GUI features
Verifying input training data is valid ... success
Copying over model files and creating run scripts ... success
A new directory has been created: /home/ubuntu/model
In this directory are 3 directories 1fm,3fm,5fm which
correspond to 3 caffe models that need to be trained
Latest iteration found in 1fm from /home/ubuntu/sbem/mitochrondria/xy5.9nm40nmz/30000iterations_train_out is 30000
Adding 20 iterations so will now run to 30020 iterations
Copying over trained models
Copy of /home/ubuntu/sbem/mitochrondria/xy5.9nm40nmz/30000iterations_train_out/1fm/trainedmodel to /home/ubuntu/model/1fm/trainedmodel success
Copy of /home/ubuntu/sbem/mitochrondria/xy5.9nm40nmz/30000iterations_train_out/3fm/trainedmodel to /home/ubuntu/model/3fm/trainedmodel success
Copy of /home/ubuntu/sbem/mitochrondria/xy5.9nm40nmz/30000iterations_train_out/5fm/trainedmodel to /home/ubuntu/model/5fm/trainedmodel success
Single GPU detected.
Resuming run from snapshot file: /home/ubuntu/model/1fm/trainedmodel/1fm_classifer_iter_30100.solverstate
Resuming run from snapshot file: /home/ubuntu/model/3fm/trainedmodel/3fm_classifer_iter_30100.solverstate
Resuming run from snapshot file: /home/ubuntu/model/5fm/trainedmodel/5fm_classifer_iter_30100.solverstate
Resuming run from snapshot file: /home/ubuntu/model/1fm/trainedmodel/1fm_classifer_iter_30100.solverstate
Resuming run from snapshot file: /home/ubuntu/model/3fm/trainedmodel/3fm_classifer_iter_30100.solverstate
Resuming run from snapshot file: /home/ubuntu/model/5fm/trainedmodel/5fm_classifer_iter_30100.solverstate
Training has completed. Have a nice day!
Training has completed. Results are stored in /home/ubuntu/model
Have a nice day!
C. Detach screen from CDeep3M training job. This can be done by hitting the key combination Control-a then the d key or in shorter form: Ctrl-a d
D. Re-attach to screen running CDeep3M training job by typing screen -r as seen here:
screen -rOnce training has finished go to the next step. A completed train job will have output text as seen above.
A. Exit from CDeep3M instance by typing exit. exit will need to be done twice since we are in a screen virtual terminal.
exit
exit
B. Download the trained model by using the scp command as seen here:
scp -i ~/.ssh/id_rsa -r ubuntu@<PublicDNS value from AWS>:/home/ubuntu/model .
ls
Congratulations on completing Tutorial 2
Click here to continue with Tutorial 3: Run CDeep3M
NOTE: If you are not continuing to the next tutorial be sure to shutdown your CDeep3M to avoid incurring additional EC2 charges.