PV021 project | Deep Learning from Scratch

[DEADLINE] Monday - 6th December 2021 (8 weeks)

[TASK]

1. Implement a feed-forward neural network in C/C++.
2. Train it on a supplied Fashion-MNIST dataset using a backpropagation algorithm.

[REQUIREMENTS] Your solution must meet ALL of the following requirements:

1. Your solution must be compilable and runnable on the AISA server.
2. Your solution achieves at least 88% accuracy.
3. Your solution must finish within 30 minutes. (parse inputs, train, evaluate, export results.)
4. Your solution must contain a runnable script called "RUN" (not run, not RUN.sh, not RUN.exe etc) which compiles, executes and exports the results into a files.
5. Your solution must output two files:
   • "trainPredictions" - network predictions for training input vectors
   • "actualTestPredictions" - network predictions for testing input vectors The format of these files is the same as the supplied training/testing labels:
   • One prediction per line.
   • Prediction for i-th input vector (ordered by the input .csv file) must be on i-th line in the associated output file.
   • Each prediction is a single integer 0 - 9.

[SCORING]

1. All submitted source files will be checked manually.
2. All submitted solutions will also be checked by an automatic evaluator. A similar evaluator is provided to you in the project folder.
3. All submitted solutions will be checked for plagiarism.
4. Any implementation that uses testing input vectors for anything else than evaluation will be awarded 0 points.
5. Presenting someone else's solution (e.g. publicly available solutions on internet) as your own will be awarded 0 points.
6. Use of high-level third-party libraries allowing matrix operations, neural network operations, differentiation operations, linear algebra operations etc is strictly forbidden. Use of such libraries will be awarded 0 points. (Low-level math operations: sqrt, exp, log, rand... is fine)

[DATASET] Fashion MNIST (https://arxiv.org/pdf/1708.07747.pdf) a modern version of a well-known MNIST (http://yann.lecun.com/exdb/mnist/). It is a dataset of Zalando's article images ‒ consisting of a training set of 60,000 examples and a test set of 10,000 examples. Each example is a 28x28 grayscale image, associated with a label from 10 classes. The dataset is in CSV format. There are four data files included: - fashion_mnist_train_vectors.csv - training input vectors - fashion_mnist_test_vectors.csv - testing input vectors - fashion_mnist_train_labels.csv - training labels - fashion_mnist_test_labels.csv - testing labels

[REMARKS]

1. What you do internally with the training dataset is up to you.
2. Write doc-strings.
3. Pack all data with your implementations and put them on the right path so your program will load them correctly on AISA (project dir is fine).
4. You can work alone or you can make teams of two.
5. There are limited number of open spots for solutions written in JAVA. You can enquire about available spots via email (422328@mail.muni.cz). Permissions will be given on a first-come first-served basis. You CANNOT submit solution in JAVA unless you have explicit permission first.

[TIPS]

1. Do not wait until the week before the deadline!
2. Consider using the suggested folder structure (data, src, etc.)
3. Execute your RUN script on AISA before your submission. Missing or non-functional RUN script cannot be evaluated.
4. Do NOT shuffle testing data. It won't fit expected predictions.
5. Solve the XOR problem first. XOR is a very nice example as a benchmark of the working learning process with at least one hidden layer. Btw, the presented network solving XOR in the lecture is minimal and it can be hard to find, so consider more neurons in the hidden layer. If you can't solve the XOR problem, you can't solve Fashion-MNIST.
6. Reuse memory. You are implementing an iterative process, so don't allocate new vectors and matrices all the time. An immutable approach is nice but very inappropriate. Also ‒ don't copy data in some cache all the time; use indexes.
7. Objects are fine, but be careful about the depth of object hierarchy you are going to create. Always remember that you are trying to be fast.
8. Double precision is fine. You may try to use floats. Do not use BigDecimals or any other high precision objects.
9. Simple SGD is not strong, and fast enough, you are going to need to implement some heuristics as well (or maybe not, but it's highly recommended). I suggest heuristics: momentum, weight decay, dropout. If you are brave enough, you can try RMSProp/AdaGrad/Adam.
10. Start with smaller networks and increase network topology carefully.
11. Consider validation of the model using part of the TRAINING dataset.
12. Adjust hyperparameters to increase your internal validation accuracy.
13. DO NOT WAIT UNTIL THE WEEK BEFORE THE DEADLINE!

[AISA]

1. .exe files are not runnable on Aisa.
2. Aisa runs on "Red Hat Enterprise Linux Server release 7.5 (Maipo)"
3. Aisa has 4×16 cores, OpenMP or similar easy parallelism may help (in case you use it, please leave some cores to other applications ~ use less than 49 cores)
4. If you are having a problem with missing compilers/maven on Aisa, you can add such tools by including modules (https://www.fi.muni.cz/tech/unix/modules.html.en). Please, do note, that if your implementation requires such modules, your RUN script must include them as well, otherwise, the RUN script won't work, and I will have no clue what to include.

[FAQ] [Q] Can I write in Python, please, please, pretty please? [A] No. It's too slow without matrix libs.

[Q] Can I implement a convolutional neural network instead of the feed-forward neural network? [A] Yes, but it might be much harder.

[Q] Can Java implementations compete with C implementations performance-wise? [A] Yes. At least one of the best performing implementations was written in java.

Best luck with the project!

Matej Gallo 422328@mail.muni.cz PV021 Neural Networks