My goal is to provide high quality Scratch Implementations of the fundamentals of Deep Learning and its applications, with interactive well documentated jupyter notebooks. All notebooks come along with implementations using Tensorflow, MXNet and Pytorch.
- It helps us to understand the core working of an algorithm better.
- We could try to improve an existing algorithm both for accuracy and efficiency.
- We could implement our own version of an existing algorithm with some new features via experiments.
- We could even invent new algorithms that could outperform previous algorithms achieving SOTA(state of the art) results.
- Thus we could actually contribute to the field.
Watch https://youtu.be/l9RWTMNnvi4?t=130, The God Father of AI Geoffrey Hinton said "If you want to understand a really complicated device like brain, then you should build one".
Same goes for the Deep Learning technologies, if you really (like reeaally) want to understand such technologies then you should build them. Indeed Keras exists, but these kind of frameworks which takes not more than 5 lines to build a model actually hides everything from you, that makes you completely unknown what's going on under the hood. The purpose of such frameworks is rapid deployment and production, they are not meant to be used by people who are beginner in the field. If you already understand the technology, then be my guest and use such frameworks like a champ, otherwise if you are a beginner in the field......... you better listen to The God Father.
- Understanding Percepron - Linear Regression
- Binary Classification - Logistic Regression
- Multiclass Classification - Softmax Regression
- The Modern Deep Learning - Neural Networks
- Sigmoid
- Tanh
- ArcTan
- Softsign
- ISRU
- ISRLU
- PLU
- ReLU
- SoftPlus
- Leaky ReLU
- PReLU
- RReLU
- ELU
- SELU
- Swish
- Bent Identity
- Momentum Optimizer
- Nesterov Accelerated Gradient Optimizer
- AdaGrad Optimizer
- AdaDelta Optimizer
- RMSProp Optimizer
- Adam Optimizer
- AMSGrad Optimizer
- Nadam Optimizer
- AdaMax Optimizer
- Recurrent Neural Networks (RNN)
- Long Short Term Memory Cell (LSTM)
- Gated Recurrent Unit (GRU)
- Sequence to Sequence
- Multilayer Recurrent Neural Networks
- Bidirectional Recurrent Neural Networks
- Understanding Convolutional
- Convolution with Padding
- Convolution with Strides
- Max Pooling
- Average Pooling
- Understanding Deconvolution
- Deconvolution with Strides
- Deconvolution with Padding
- Autoencoder
- Denoising Autoencoder
- Sparse Autoencoder
- Variational Autoencoder
- Generative Adversarial Networks
- Deep Convolutional GAN
- Conditional GAN
- AC GAN
- LS GAN
- Info GAN
- Cyclic GAN. (To be implemented)
- Disco GAN. (To be implemented)
- Bi GAN. (To be implemented)
- W GAN. (To be implemented)
- Image Classification
- Transfer Learning
- Image Denoising
- Image Resolution Inhancer. (To be implemented)
- Image Reconstruction. (To be implemented)
- Image Colorization. (To be implemented)
- Object Detection. (To be implemented)
- Objection Segmentation. (To be implemented)
- Style Transfer. (To be implemented)
- Text to Image. (To be implemented)
- Image Captioning. (To be implemented)
- Word Embeddings. (To be implemented)
- Embedding Matrix. (To be implemented)
- Word to Vec. (To be implemented)
- GloVe. (To be implemented)
- Negative Sampling. (To be implemented)
- Attention Mechanism. (To be implemented)
- Sentiment Analysis. (To be implemented)
- Named Entity Recognition. (To be implemented)
- Machine Translation. (To be implemented)
- Text Summarization. (To be implemented)
- Visual Question Answering. (To be implemented)
- Image Captioning. (To be implemented)
- Music Generation. (To be implemented)
It is recommended to run notebooks locally on your computer only if you have GPU(CUDA) support, otherwise it'd be quit painfull :O
If you don't have the GPU, then follow the steps as bellow:-
- Go to https://colab.research.google.com
- Sign In using your google account.
- Clone the repository by running
git clone - From the menu top left, Go to "File -> Upload notebook -> Choose File".
- Then select the notebook you want to run located in your computer and click "Open".
- Wait for a while.
- When notebook is successfully loaded, from the menu top left go to "Runtime -> Change runtime type".
- Select GPU as your "Hardware accelerator".
- Click "SAVE".
- Now you are ready to roll.
Here are some very useful resources I've curated so far. First we will learn some basic fundamentals of Deep Learning, then we will learn the core math required to understand a bit complex Deep Learning approaches, so watch and read in the following order.
-
Deep Learning For Beginners by Andrew Ng
Its a full course for beginners by Prof Andrew Ng, this course provides the glimps to the world of AI.
-
Deep Learning For Intermediates by Andrew Ng
Its a full course for intermediates by Prof Andrew Ng, this course provides techniques to improve Deep Learning models.
-
Convolutional Neural Networks by Andrew Ng
Its a full course on Convolutional Neural Networks by Prof Andrew Ng, this course provides great explanations and examples for the topic with great simplicity and helpful guides.
-
Recurrent Neural Networks by Andrew Ng
Its a full course on Recurrent Neural Networks by Prof Andrew Ng, this course provides explanations to why we need sequence models and what are their use cases and how do they work.
-
An Outstanding course for Linear Algebra by Prof W. Gilbert Strange
This course is by far my best course on Linear Algebra on the planet. Prof Strange has done a phenomenal job, just watch his videos and I bet you'd see a rainbow with a unicorn.
-
An Another Outstanding Course on Linear Algera By 3Blue1Brown
This course introduces the "Essence of linear algebra" series, aimed at animating the geometric intuitions underlying many of the topics taught in a standard linear algebra course.
-
Course on Calculus by 3Blue1Brown
What might it feel like to invent calculus?
-
A Great Course on Deep Learning by MIT
Its a full course on Deep Learning by MIT, this course provides great explanations to various deep learning topics like GANs... etc. Note: only watch first 4 lectures because afterwards they are explaining an another AI field know as Reinforcement Learning, which we will cover later.
-
Deep Learning Course by Stanford
This course covers introduction to deep learning and its applications like Adversarial Attacks, Interpretability and Health Care... etc. Note: skip the Reinforcement Learning part for now, we will cover that later.
-
An Awesome Book on Deep Learning
This book is an interactive deep learning book with codes, maths, and discussions, for beginners as well as for experts. It provides crystal clear explaination for all the topics along with hands on coding tutorials. If you got question regarding the topic then you can ask it to the forum as well.
-
Another Great Book on Deep Learning by Ian Goodfellow
This book introduces to Deep Learning by first teaching the crucial math required for research in Deep Learning, for example:-
- Linear Algebra
- Probability
- Numerical Computation... etc
This book would definitely help you to build the foundational understanding of mathematics for research in Deep Learning.
- Implement all above algorithms from scratch in Tensorflow.
- Implement all above algorithms from scratch in Pytorch.
- Provide resources to learn topics mentioned above.
- Add more explanations to notebooks for better understanding.