Deep-Learning-Systems

CMU10714 - Deep Learning Systems: Algorithms and Implementation

Implementation courses

• Lec5: Automatic Differentiation
• Lec8: Neural Network Library
• Lec13: Hardware Acceleration
• Lec14: Convolution Network Implementation
• Lec17: Generative Adversarial Networks
• Lec19: Recurrent Network
• Lec21:Transformers

Homework Summary

• Hw0: prior knowledge review

  • cross-entropy loss
  • SGD
  • softmax regression
  • two-layer nn

• Hw1: build a basic automatic differentiation framework

  • backprogation
  • topological sort
  • reverse mode automatic differentiation

• Hw2: implement a neural network library in the needle framework

  • Weight initialization: Xavier and Kaiming
  • Modules: Linear, ReLu, Sequentail, LogSumExp, SoftmaxLoss, Normalizaiton(Layer/Batch), Flatten, Dropout, Residual
  • Optimizers: SGD, Adam
  • Data primitives: Dataloader and Dataset
  • Build and train MLP ResNet

• Hw3: building an NDArray library