|
38 | 38 | # |
39 | 39 | # The optimziation loop is comprized of three main subloops in PyTorch. |
40 | 40 | # |
41 | | -# .. figure:: /_static/img/quickstart/optimization_loops.png |
| 41 | + |
| 42 | +############################################################ |
| 43 | +# .. figure:: /_static/img/quickstart/optimizationloops.png |
42 | 44 | # :alt: |
43 | 45 | # |
44 | | -# |
| 46 | + |
| 47 | +############################################################# |
45 | 48 | # 1. The Train Loop - Core loop iterates over all the epochs |
46 | 49 | # 2. The Validation Loop - Validate loss after each weight parameter update and can be used to gauge hyper parameter performance and update them for the next batch. |
47 | 50 | # 3. The Test Loop - is used to evaluate our models performance after each epoch on traditional metrics to show how much our model is generalizing from the train and validation dataset to the test dataset it's never seen before. |
48 | 51 | # |
49 | 52 |
|
50 | | -for epoch in range(num_epochs): # Optimization Loop |
| 53 | +for epoch in range(num_epochs): |
| 54 | +# Optimization Loop |
51 | 55 | # Train loop over batches |
52 | | - model.train() # set model to train |
53 | | - # Model Update Code |
54 | | - model.eval() # After exiting batch loop set model to eval to speed up evaluation and not track gradients (this is explained below) |
55 | | - # Validation Loop |
56 | | - # - Put sample validation metric logging and hyperparameter update code here |
| 56 | + model.train() # set model to train |
| 57 | + # Model Update Code |
| 58 | + model.eval() # After exiting batch loop set model to eval to speed up evaluation and not track gradients (this is explained below) |
| 59 | + # Validation Loop |
| 60 | + # - Put sample validation metric logging and hyperparameter update code here |
57 | 61 | # After exiting train loop set model to eval to speed up evaluation and not track gradients (this is explained below) |
58 | 62 | # Test Loop |
59 | | - # - Put sample test metric logging and hyperparameter update code here |
| 63 | + # - Put sample test metric logging and hyperparameter update code here |
60 | 64 |
|
61 | 65 | ###################################################### |
62 | 66 | # Loss |
|
67 | 71 |
|
68 | 72 | preds = model(inputs) |
69 | 73 | loss = cost_function(preds, labels) |
70 | | - |
71 | | -###################################################### |
72 | | -# AutoGrad and Optimizer (We might want to split this when we go more in depth on autograd ) |
73 | | -# ----------------- |
74 | | -# |
75 | | -# By default each tensor maintains a graph of every operation applied on it unless otherwise specified using the torch.no_grad() command. |
76 | | -# |
77 | | -# `Autograd graph <https://discuss.pytorch.org/uploads/default/original/1X/c7e0a44b7bcebfb41315b56f8418ce37f0adbfeb.png>`_ |
78 | | -# |
79 | | -# PyTorch uses this graph to automatically update parameters with respect to our models loss during training. This is done with one line loss.backwards(). Once we have our gradients the optimizer is used to propgate the gradients from the backwards command to update all the parameters in our model. |
80 | | - |
81 | | -optimizer.zero_grad() # make sure previous gradients are cleared |
82 | | -loss.backward() # calculates gradients with respect to loss |
| 74 | +# Make sure previous gradients are cleared |
| 75 | +optimizer.zero_grad() |
| 76 | +# Calculates gradients with respect to loss |
| 77 | +loss.backward() |
83 | 78 | optimizer.step() |
84 | 79 |
|
85 | 80 | ###################################################### |
86 | | -# The standard method for optimization is called Stochastic Gradient Descent, to learn more check out this awesome video by `3blue1brown <https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi>`_. There are many different optimizers and variations of this method in PyTorch such as ADAM and RMSProp that work better for different kinds of models, they are out side the scope of this Blitz, but can check out the full list of optimizers[here](https://pytorch.org/docs/stable/optim.html) |
| 81 | +# The standard method for optimization is called Stochastic Gradient Descent, to learn more check out this awesome video by `3blue1brown <https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi>`_. There are many different optimizers and variations of this method in PyTorch such as ADAM and RMSProp that work better for different kinds of models, they are out side the scope of this Blitz, but can check out the full list of optimizers `here <https://pytorch.org/docs/stable/optim.html>`_ |
87 | 82 |
|
88 | 83 | ###################################################### |
89 | 84 | # Putting it all together lets look at a basic optimization loop |
90 | 85 | # ----------------- |
91 | 86 | # |
92 | 87 | # Initilize optimizer and example cost function |
93 | 88 | # |
94 | | -# # For loop to iterate over epoch |
95 | | -# - Train loop over batches |
96 | | -# - Set model to train mode |
97 | | -# - Calculate loss using |
98 | | -# - clear optimizer gradient |
99 | | -# - loss.backword |
100 | | -# - optimizer step |
101 | | -# - Set model to evaluate mode and start validation loop |
102 | | -# - calculate validation loss and update optimizer hyper parameters |
103 | | -# - Set model to evaluate test loop |
| 89 | +# For loop to iterate over epoch |
| 90 | +# - Train loop over batches |
| 91 | +# - Set model to train mode |
| 92 | +# - Calculate loss using |
| 93 | +# - clear optimizer gradient |
| 94 | +# - loss.backword |
| 95 | +# - optimizer step |
| 96 | +# - Set model to evaluate mode and start validation loop |
| 97 | +# - calculate validation loss and update optimizer hyper parameters |
| 98 | +# - Set model to evaluate test loop |
104 | 99 |
|
105 | 100 |
|
106 | 101 | ################################################################## |
|
0 commit comments