markdown source builds

Auto-generated via {sandpaper}
Source  : 113c45d
Branch  : main
Author  : Jonathan Taylor <jonathan.taylor@manchester.ac.uk>
Time    : 2023-09-13 15:49:28 +0000
Message : Add remaining ep4 material

04-create_net.md

@@ -44,13 +44,159 @@ From a high level, a neural network is a system that takes input values in an 
 
 ![](fig/simple_neural_network.png){a;t="A simple neural network with input, output, and 2 hidden layers"}
 
+The layers shown in the network above are "dense" or "fully connected". Each neuron is connected to all neurons in the preceeding layer. Dense layers are a common building block in neural network architectures.
 
+“Deep learning” is an increasingly popular term used to describe certain types of neural network. When people talk about deep learning they are typically referring to more complex network designs, often with a large number of hidden layers.
 
+## Activation Functions
 
+Part of the concept of a neural network is that each neuron can either be 'active' or 'inactive'. This notion of activity and inactivity is attempted to be replicated by so called activation functions. The original activation function was the sigmoid function (related to its use in logistic regression). This would make each neuron's activation some number between 0 and 1, with the idea that 0 was 'inactive' and 1 was 'active'.
 
+As time went on, different activation functions were used. For example the tanh function (hyperbolic tangent function), where the idea is a neuron can be active in both a positive capacity (close to 1), a negative capacity (close to -1) or can be inactive (close to 0).
 
+The problem with both of these is that they suffered from a problem called [model saturation](http://vigir.missouri.edu/~gdesouza/Research/Conference_CDs/IEEE_SSCI_2015/data/7560b423.pdf). This is where very high or very low values are put into the activation function, where the gradient of the line is almost flat. This leads to very slow learning rates (it can take a long time to train models with these activation functions).
 
+Another very popular activation function that tries to tackle this is the rectified linear unit (ReLU) function. This has 0 if the input is negative (inactive) and just gives back the input if it is positive (a measure of how active it is - the metaphor gets rather stretched here). This is much faster at training and gives very good performance, but still suffers model saturation on the negative side. Researchers have tried to get round this with functions like 'leaky' ReLU, where instead of returning 0, negative inputs are multiplied by a very small number.
 
+![](fig/ActivationFunctions.png){alt="Plots of the Sigmoid, Tanh, ReLU, and Leaky ReLU activation functions"}
+
+## Convolutional neural networks
+
+Convolutional neural networks (CNNs) are a type of neural network that especially popular for vision tasks such as image recognition. CNNs are very similar to ordinary neural networks, but they have characteristics that make them well suited to image processing.
+
+Just like other neural networks, a CNN typically consists of an input layer, hidden layers and an output layer. The layers of "neurons" have learnable weights and biases, just like other networks.
+
+What makes CNNs special? The name stems from the fact that the architecture includes one or more convolutional layers. These layers apply a mathematical operation called a "convolution" to extract features from arrays such as images.
+
+In a convolutional layer, a matrix of values referred to as a "filter" or "kernel" slides across the input matrix (in our case, an image). As it slides, values are multiplied to generate a new set of values referred to as a "feature map" or "activation map".
+
+![](fig/convolution_plotke.gif){alt="2D Convolution Animation by Michael Plotke"}
+
+Filters provide a mechanism for emphasising aspects of an input image. For example, a filter may emphasise object edges. See [setosa.io](https://setosa.io/ev/image-kernels/) for a visual demonstration of the effect of different filters.
+
+## Creating a convolutional neural network
+
+Before training a convolutional neural network, we will first need to define the architecture. We can do this using the Keras and Tensorflow libraries.
+
+```python
+# Create the architecture of our convolutional neural network, using
+# the tensorflow library
+from tensorflow.random import set_seed
+from tensorflow.keras.layers import Dense, Dropout, Conv2D, MaxPool2D, Input, GlobalAveragePooling2D
+from tensorflow.keras.models import Model
+
+# set random seed for reproducibility
+set_seed(42)
+
+# Our input layer should match the input shape of our images.
+# A CNN takes tensors of shape (image_height, image_width, color_channels)
+# We ignore the batch size when describing the input layer
+# Our input images are 256 by 256, plus a single colour channel.
+inputs = Input(shape=(256, 256, 1))
+
+# Let's add the first convolutional layer
+x = Conv2D(filters=8, kernel_size=3, padding='same', activation='relu')(inputs)
+
+# MaxPool layers are similar to convolution layers. 
+# The pooling operation involves sliding a two-dimensional filter over each channel of feature map and selecting the max values.
+# We do this to reduce the dimensions of the feature maps, helping to limit the amount of computation done by the network.
+x = MaxPool2D()(x)
+
+# We will add more convolutional layers, followed by MaxPool
+x = Conv2D(filters=8, kernel_size=3, padding='same', activation='relu')(x)
+x = MaxPool2D()(x)
+x = Conv2D(filters=12, kernel_size=3, padding='same', activation='relu')(x)
+x = MaxPool2D()(x)
+x = Conv2D(filters=12, kernel_size=3, padding='same', activation='relu')(x)
+x = MaxPool2D()(x)
+x = Conv2D(filters=20, kernel_size=5, padding='same', activation='relu')(x)
+x = MaxPool2D()(x)
+x = Conv2D(filters=20, kernel_size=5, padding='same', activation='relu')(x)
+x = MaxPool2D()(x)
+x = Conv2D(filters=50, kernel_size=5, padding='same', activation='relu')(x)
+
+# Global max pooling reduces dimensions back to the input size
+x = GlobalAveragePooling2D()(x)
+
+# Finally we will add two "dense" or "fully connected layers".
+# Dense layers help with the classification task, after features are extracted.
+x = Dense(128, activation='relu')(x)
+
+# Dropout is a technique to help prevent overfitting that involves deleting neurons.
+x = Dropout(0.6)(x)
+
+x = Dense(32, activation='relu')(x)
+
+# Our final dense layer has a single output to match the output classes.
+# If we had multi-classes we would match this number to the number of classes.
+outputs = Dense(1, activation='sigmoid')(x)
+
+# Finally, we will define our network with the input and output of the network
+model = Model(inputs=inputs, outputs=outputs)
+```
+
+We can view the architecture of the model:
+
+```python
+model.summary()
+```
+
+```output
+Model: "model_39"
+_________________________________________________________________
+ Layer (type)                Output Shape              Param #   
+=================================================================
+ input_9 (InputLayer)        [(None, 256, 256, 1)]     0         
+                                                                 
+ conv2d_59 (Conv2D)          (None, 256, 256, 8)       80        
+                                                                 
+ max_pooling2d_50 (MaxPoolin  (None, 128, 128, 8)      0         
+ g2D)                                                            
+                                                                 
+ conv2d_60 (Conv2D)          (None, 128, 128, 8)       584       
+                                                                 
+ max_pooling2d_51 (MaxPoolin  (None, 64, 64, 8)        0         
+ g2D)                                                            
+                                                                 
+ conv2d_61 (Conv2D)          (None, 64, 64, 12)        876       
+                                                                 
+ max_pooling2d_52 (MaxPoolin  (None, 32, 32, 12)       0         
+ g2D)                                                            
+                                                                 
+ conv2d_62 (Conv2D)          (None, 32, 32, 12)        1308      
+                                                                 
+ max_pooling2d_53 (MaxPoolin  (None, 16, 16, 12)       0         
+ g2D)                                                            
+                                                                 
+ conv2d_63 (Conv2D)          (None, 16, 16, 20)        6020      
+                                                                 
+ max_pooling2d_54 (MaxPoolin  (None, 8, 8, 20)         0         
+ g2D)                                                            
+                                                                 
+ conv2d_64 (Conv2D)          (None, 8, 8, 20)          10020     
+                                                                 
+ max_pooling2d_55 (MaxPoolin  (None, 4, 4, 20)         0         
+ g2D)                                                            
+                                                                 
+ conv2d_65 (Conv2D)          (None, 4, 4, 50)          25050     
+                                                                 
+ global_average_pooling2d_8   (None, 50)               0         
+ (GlobalAveragePooling2D)                                        
+                                                                 
+ dense_26 (Dense)            (None, 128)               6528      
+                                                                 
+ dropout_8 (Dropout)         (None, 128)               0         
+                                                                 
+ dense_27 (Dense)            (None, 32)                4128      
+                                                                 
+ dense_28 (Dense)            (None, 1)                 33        
+                                                                 
+=================================================================
+Total params: 54,627
+Trainable params: 54,627
+Non-trainable params: 0
+_________________________________________________________________
+```
 
 
 

md5sum.txt

@@ -6,7 +6,7 @@
 "episodes/01-xrays.md" "3f632d7f57ae4500685f33084569b2e6" "site/built/01-xrays.md" "2023-09-07"
 "episodes/02-visualisation.md" "a4691efb3b405086c3e4c15e2162ca56" "site/built/02-visualisation.md" "2023-09-11"
 "episodes/03-data_prep.md" "c91610c10d0869777b70634a41a3e41c" "site/built/03-data_prep.md" "2023-09-11"
-"episodes/04-create_net.md" "6a53a0fcde1e1e6cc1d5c2e9d8a98761" "site/built/04-create_net.md" "2023-09-13"
+"episodes/04-create_net.md" "35b1f4625a4a60055b4ecece09c7d847" "site/built/04-create_net.md" "2023-09-13"
 "episodes/05-scripts.md" "345b218b165e690c05526faa0fd9e276" "site/built/05-scripts.md" "2023-08-21"
 "episodes/06-func.md" "3124afd9a5a2d2434c4e4d34056842cc" "site/built/06-func.md" "2023-08-21"
 "episodes/07-loops.md" "85de3b8735b0c68d274bb989c33c0f81" "site/built/07-loops.md" "2023-08-21"