Math and Architectures of Deep Learning

Python code (in the form of Jupyter ipython notebooks) to support the book "Math and Architectures of Deep Learning".

Code contributors: Ananya Ashok, Sujay Narumanchi, Devashish Shankar, Krishnendu Chaudhury.

This repository contains the example code - mostly in Numpy and PyTorch - corresponding to the theoretical topics introduced in the book. The code listings are organized in chapters that correspond to the main book.

Installation

1. Clone the repository: git clone https://github.com/krishnonwork/mathematical-methods-in-deep-learning-ipython.git
2. Create virtual environment: virtualenv venv --python=python3 (you may need to do pip install virtualenv first)
3. Activate virtual environment: source venv/bin/activate
4. Change directory: cd mathematical-methods-in-deep-learning-ipython
5. Install dependencies: pip install -r requirements.txt
6. Navigate to the python directory: cd python
7. Start jupyter: jupyter notebook

This will redirect you to a browser window with the ipython notebooks

Note: Ensure to use Python3 to run the notebooks

Table of Contents

• Chapter 2:

  • 2.2 Intro to Vectors
  • 2.4 Intro to Matrices, Tensors and Images
  • 2.7 Basic Vector and Matrix operations
  • 2.12.5 Solving an overdetermined system using pseudo inverse
  • 2.13 Eigenvalues and Eigenvectors
  • 2.14 Rotation Matrices
  • 2.15 Matrix Diagonalization
  • 2.16 Spectral Decomposition of a Symmetric Matrix
  • 2.17 Finding the axes of a hyper-ellipse

• Chapter 3

  • 3.4 Common code for chapter 3
  • 3.4.1 Gradient Descent
  • 3.4.2 Non-linear Models
  • 3.4.3 A Linear Model for the cat-brain

• Chapter 4

  • 4.3.2 Common code for chapter 4
  • 4.3.2 PCA on synthetic correlated data
  • 4.3.2 PCA on synthetic uncorrelated data
  • 4.3.3 PCA on synthetic correlated non-linear data
  • 4.4.4 Linear system solving via SVD
  • 4.4.5 PCA computation via SVD
  • 4.5.3 LSA on a toy dataset
  • 4.5.4 LSA/SVD on a 500 × 3 dataset

• Chapter 5

  • 5.9.1 Uniform Random Distribution
  • 5.9.2 Normal Distribution
  • 5.9.3 Binomial Distribution
  • 5.9.4 Multinomial Distribution
  • 5.9.5 Bernoulli Distribution

• Chapter 6

  • 6.2.2 Entropy
  • 6.3.1 Cross Entropy
  • 6.4.2 KL Divergence
  • 6.7.1 Model Parameter Estimation
  • 6.8.2 Gaussian Mixture Modelling

• Chapter 7

  • 7.2.3 Perceptron
  • 7.2.4 Modeling logic gates with perceptrons
  • 7.4.3 Approximating surfaces with perceptrons

• Chapter 8

  • 8.4 Forward and Backward pass
  • 8.5 Training a neural network

• Chapter 9

  • 9.1 Loss Functions
  • 9.2.3 Stochastic Gradient Descent
  • 9.2.4 Momentum

• Chapter 10

  • 10.3.1 2D Convolution for image smoothing
  • 10.3.2 2D Convolution for edge detection
  • 10.4.1 3D Convolution for motion detection
  • 10.5.3 Transpose Convolution for upsampling

• Chapter 11

  • 11.2.1 Convolutional Neural Networks for Image Classification - LeNet
  • 11.3.1 VGG (Visual Geometry Group) Net
  • 11.3.2 Inception: Network in Network paradigm
  • 11.3.3 ResNet
  • 11.3.4 PyTorch Lightning: LeNet
  • 11.3.4 PyTorch Lightning: Deep CNN Classifier
  • 11.5 Faster R-CNN

• Chapter 13

  • 13.2 Bayesian Inferencing for mean in Gaussian likelihood, known variance
  • 13.5.1 Gamma Distribution
  • 13.5 Bayesian Inferencing of Precision of Gaussian likelihood, known Mean
  • 13.6 Bayesian Inferencing of both Mean and Precision of Gaussian likelihood
  • 13.7 Statsville Revisited: Bayesian inferencing to predict if a Statsville resident is female based on height
  • 13.8 Multivariate Bayesian Inferencing of Mean of Gaussian likelihood, known Precision

• Chapter 14

  • 14.1 Principal Components Analysis (PCA) recap
  • 14.2 Auto Encoders
  • 14.4 Variational Auto Encoders