Fix MNIST example not training (pointer to de-referenced stack memory)

[MadLittleMods]

Fix MNIST example not training

We were referencing a stack allocated variable in the loop which just went to garbled memory after the loop ended (happened to reference the last item of the loop).

It would be nice if Zig could warn/error when we tried to reference and store a pointer to some stack-allocated object to avoid this type footgun (ziglang/zig#2301, ziglang/zig#5725, ziglang/zig#2646, https://www.openmymind.net/Zig-Danling-Pointers/).

Before

Before we fixed this bug, every labeled piece of data was using the image of the 8 (whatever the last training item happens to be).

(Notice how it's showing an 8 but it's labeled as 5 ❌)

$ zig build run-mnist_ocr
debug: training labels header mnist.mnist_data_utils.MnistLabelFileHeader{ .magic_number = 2049, .number_of_labels = 60000 }
debug: training images header mnist.mnist_data_utils.MnistImageFileHeader{ .magic_number = 2051, .number_of_images = 60000, .number_of_rows = 28, .number_of_columns = 28 }
debug: testing labels header mnist.mnist_data_utils.MnistLabelFileHeader{ .magic_number = 2049, .number_of_labels = 10000 }
debug: testing images header mnist.mnist_data_utils.MnistImageFileHeader{ .magic_number = 2051, .number_of_images = 10000, .number_of_rows = 28, .number_of_columns = 28 }
debug: Created normalized data points. Training on 60000 data points, testing on 100
┌──────────┐
│ Label: 5 │
┌────────────────────────────────────────────────────────┐
│                                                        │
│                                                        │
│                                                        │
│                                                        │
│                                                        │
│                                                        │
│                                ░░░░░░░░                │
│                          ░░▒▒██████████░░              │
│                        ▒▒▓▓██████████████░░  ░░▓▓▓▓    │
│                      ▒▒████▒▒▒▒░░    ▒▒██▒▒░░████░░    │
│                    ░░████▒▒░░        ░░▓▓██████▒▒      │
│                    ▒▒██▒▒            ▒▒████▓▓░░        │
│                    ▓▓██░░      ░░▒▒██████▒▒░░          │
│                    ██▒▒    ░░░░▓▓████▒▒                │
│                    ▓▓██░░░░▒▒████▓▓░░                  │
│                    ▓▓██▒▒▓▓██▓▓░░                      │
│                    ▓▓██████░░                          │
│                ░░▒▒████▓▓░░                            │
│                ▒▒██████▒▒                              │
│              ▓▓██▓▓░░██▒▒                              │
│            ▒▒██▓▓░░░░██░░                              │
│            ▓▓██▓▓  ░░██▓▓                              │
│            ▒▒██▓▓  ░░██▒▒                              │
│            ░░████▓▓████░░                              │
│              ▒▒████▓▓                                  │
│                                                        │
│                                                        │
│                                                        │
└────────────────────────────────────────────────────────┘

After

Now all of the images correctly line up with their labels ✅

$ zig build run-mnist_ocr
debug: training labels header mnist.mnist_data_utils.MnistLabelFileHeader{ .magic_number = 2049, .number_of_labels = 60000 }
debug: training images header mnist.mnist_data_utils.MnistImageFileHeader{ .magic_number = 2051, .number_of_images = 60000, .number_of_rows = 28, .number_of_columns = 28 }
debug: testing labels header mnist.mnist_data_utils.MnistLabelFileHeader{ .magic_number = 2049, .number_of_labels = 10000 }
debug: testing images header mnist.mnist_data_utils.MnistImageFileHeader{ .magic_number = 2051, .number_of_images = 10000, .number_of_rows = 28, .number_of_columns = 28 }
debug: Created normalized data points. Training on 60000 data points, testing on 100
┌──────────┐
│ Label: 5 │
┌────────────────────────────────────────────────────────┐
│                                                        │
│                                                        │
│                                                        │
│                                                        │
│                                                        │
│                        ░░░░░░░░▒▒▓▓▓▓░░▓▓████▒▒        │
│                ░░░░▒▒▓▓▓▓████████████▓▓██████▒▒        │
│              ░░████████████████████▒▒▒▒▒▒░░░░          │
│              ░░██████████████▓▓████                    │
│                ▒▒▓▓▒▒██████░░  ░░▓▓                    │
│                  ░░░░▓▓██▒▒                            │
│                      ▓▓██▓▓░░                          │
│                      ░░▓▓██▒▒                          │
│                        ░░████▓▓▒▒░░                    │
│                          ▒▒██████▒▒░░                  │
│                            ░░▓▓████▓▓░░                │
│                              ░░▒▒████▓▓                │
│                                  ██████▒▒              │
│                            ░░▓▓▓▓██████░░              │
│                        ░░▓▓██████████▓▓                │
│                    ░░▒▒████████████▒▒                  │
│                ░░▒▒████████████▒▒░░                    │
│            ░░▓▓████████████▒▒░░                        │
│        ░░▓▓████████████▓▓░░                            │
│        ▓▓████████▓▓▓▓░░                                │
│                                                        │
│                                                        │
│                                                        │
└────────────────────────────────────────────────────────┘

Now training can make progress!!

$ zig build run-mnist_ocr
debug: training labels header mnist.mnist_data_utils.MnistLabelFileHeader{ .magic_number = 2049, .number_of_labels = 60000 }
debug: training images header mnist.mnist_data_utils.MnistImageFileHeader{ .magic_number = 2051, .number_of_images = 60000, .number_of_rows = 28, .number_of_columns = 28 }
debug: testing labels header mnist.mnist_data_utils.MnistLabelFileHeader{ .magic_number = 2049, .number_of_labels = 10000 }
debug: testing images header mnist.mnist_data_utils.MnistImageFileHeader{ .magic_number = 2051, .number_of_images = 10000, .number_of_rows = 28, .number_of_columns = 28 }
debug: Created normalized data points. Training on 60000 data points, testing on 100
debug: initial layer biases { 0.100, 0.100, 0.100, 0.100, 0.100, 0.100, 0.100, 0.100, 0.100, 0.100 }
debug: epoch 0   batch 0             3s -> cost 328.2142204682694, accuracy with test points 0.11
debug: epoch 0   batch 1             3s -> cost 315.4082089553885, accuracy with test points 0.19
debug: epoch 0   batch 2             3s -> cost 300.4535044810213, accuracy with test points 0.34
debug: epoch 0   batch 3             3s -> cost 284.30505300566585, accuracy with test points 0.49
debug: epoch 0   batch 4             3s -> cost 267.61381440721016, accuracy with test points 0.57
debug: epoch 0   batch 5             3s -> cost 251.30090715210744, accuracy with test points 0.59
debug: epoch 0   batch 6             4s -> cost 233.80498092236553, accuracy with test points 0.56
debug: epoch 0   batch 7             4s -> cost 217.5972513474019, accuracy with test points 0.62
debug: epoch 0   batch 8             4s -> cost 201.33117424537568, accuracy with test points 0.65
debug: epoch 0   batch 9             4s -> cost 186.04815224585556, accuracy with test points 0.66
debug: epoch 0   batch 10            4s -> cost 173.36122442454, accuracy with test points 0.66
debug: epoch 0   batch 11            5s -> cost 161.27042938581928, accuracy with test points 0.68
debug: epoch 0   batch 12            5s -> cost 151.0171197075731, accuracy with test points 0.69
debug: epoch 0   batch 13            5s -> cost 142.0996775497194, accuracy with test points 0.71
debug: epoch 0   batch 14            5s -> cost 135.07975432834573, accuracy with test points 0.74
debug: epoch 0   batch 15            5s -> cost 129.8505114553166, accuracy with test points 0.75
debug: epoch 0   batch 16            5s -> cost 124.21138149012033, accuracy with test points 0.74
debug: epoch 0   batch 17            6s -> cost 118.41200990073645, accuracy with test points 0.76
debug: epoch 0   batch 18            6s -> cost 112.8523425063855, accuracy with test points 0.76
debug: epoch 0   batch 19            6s -> cost 108.57819166349036, accuracy with test points 0.8
debug: epoch 0   batch 20            6s -> cost 105.07134500738981, accuracy with test points 0.81
debug: epoch 0   batch 21            6s -> cost 104.7101921741118, accuracy with test points 0.77
debug: epoch 0   batch 22            6s -> cost 107.09339427534043, accuracy with test points 0.75
debug: epoch 0   batch 23            7s -> cost 108.37265298151958, accuracy with test points 0.75
debug: epoch 0   batch 24            7s -> cost 108.90795542855233, accuracy with test points 0.75
debug: epoch 0   batch 25            7s -> cost 105.9608632207419, accuracy with test points 0.77
debug: epoch 0   batch 26            7s -> cost 99.18349310993631, accuracy with test points 0.76
debug: epoch 0   batch 27            7s -> cost 88.57364151575837, accuracy with test points 0.8
debug: epoch 0   batch 28            7s -> cost 83.35502720997414, accuracy with test points 0.82
debug: epoch 0   batch 29            8s -> cost 82.01587747821182, accuracy with test points 0.84
debug: epoch 0   batch 30            8s -> cost 82.39684434985433, accuracy with test points 0.86
debug: epoch 0   batch 31            8s -> cost 82.96086077467842, accuracy with test points 0.86
debug: epoch 0   batch 32            8s -> cost 83.93685525183301, accuracy with test points 0.84
debug: epoch 0   batch 33            8s -> cost 83.72949283480301, accuracy with test points 0.83
debug: epoch 0   batch 34            8s -> cost 84.16376070458978, accuracy with test points 0.83
debug: epoch 0   batch 35            9s -> cost 83.65926200997627, accuracy with test points 0.84
debug: epoch 0   batch 36            9s -> cost 82.46338659448206, accuracy with test points 0.85
debug: epoch 0   batch 37            9s -> cost 79.42798668739822, accuracy with test points 0.86
debug: epoch 0   batch 38            9s -> cost 77.45648769512488, accuracy with test points 0.88
debug: epoch 0   batch 39            9s -> cost 75.70461541681406, accuracy with test points 0.9
debug: epoch 0   batch 40           10s -> cost 73.3612682266723, accuracy with test points 0.9
debug: epoch 0   batch 41           10s -> cost 70.60106796491523, accuracy with test points 0.91
debug: epoch 0   batch 42           10s -> cost 68.31163012909519, accuracy with test points 0.9
debug: epoch 0   batch 43           10s -> cost 68.06393899119688, accuracy with test points 0.91
debug: epoch 0   batch 44           10s -> cost 68.14894186570761, accuracy with test points 0.9
debug: epoch 0   batch 45           10s -> cost 67.128872687339, accuracy with test points 0.9
debug: epoch 0   batch 46           11s -> cost 64.33807323265755, accuracy with test points 0.91
debug: epoch 0   batch 47           11s -> cost 63.49807076617511, accuracy with test points 0.9
debug: epoch 0   batch 48           11s -> cost 65.39864471075207, accuracy with test points 0.87
debug: epoch 0   batch 49           11s -> cost 67.4759928434266, accuracy with test points 0.86
debug: epoch 0   batch 50           11s -> cost 67.1559864272731, accuracy with test points 0.87
debug: epoch 0   batch 51           11s -> cost 65.85677948175478, accuracy with test points 0.9
debug: epoch 0   batch 52           12s -> cost 63.53133614183292, accuracy with test points 0.89
debug: epoch 0   batch 53           12s -> cost 61.32981821668863, accuracy with test points 0.88
debug: epoch 0   batch 54           12s -> cost 60.321206103340806, accuracy with test points 0.89
debug: epoch 0   batch 55           12s -> cost 60.419791821748404, accuracy with test points 0.9
debug: epoch 0   batch 56           12s -> cost 59.89266251877525, accuracy with test points 0.91
debug: epoch 0   batch 57           13s -> cost 61.06359883050953, accuracy with test points 0.91
debug: epoch 0   batch 58           13s -> cost 62.92734772559315, accuracy with test points 0.92
debug: epoch 0   batch 59           13s -> cost 65.38602253708358, accuracy with test points 0.9
debug: epoch 0   batch 60           13s -> cost 68.16367181206456, accuracy with test points 0.91
debug: epoch 0   batch 61           13s -> cost 69.72994374419666, accuracy with test points 0.91
debug: epoch 0   batch 62           13s -> cost 70.40323925331515, accuracy with test points 0.92
debug: epoch 0   batch 63           14s -> cost 71.41243566368676, accuracy with test points 0.91
debug: epoch 0   batch 64           14s -> cost 72.40166411962798, accuracy with test points 0.9
debug: epoch 0   batch 65           14s -> cost 72.91261989070384, accuracy with test points 0.89
debug: epoch 0   batch 66           14s -> cost 74.01402636422786, accuracy with test points 0.88
debug: epoch 0   batch 67           14s -> cost 75.12883685428005, accuracy with test points 0.88
debug: epoch 0   batch 68           14s -> cost 73.20024013767603, accuracy with test points 0.89
debug: epoch 0   batch 69           15s -> cost 70.46390675818081, accuracy with test points 0.88
debug: epoch 0   batch 70           15s -> cost 66.56112725467558, accuracy with test points 0.88
debug: epoch 0   batch 71           15s -> cost 63.844685394807435, accuracy with test points 0.88
debug: epoch 0   batch 72           15s -> cost 60.993829097294444, accuracy with test points 0.89
debug: epoch 0   batch 73           15s -> cost 59.54636368759743, accuracy with test points 0.92
debug: epoch 0   batch 74           15s -> cost 57.992966190988454, accuracy with test points 0.91
debug: epoch 0   batch 75           16s -> cost 57.1296321510229, accuracy with test points 0.91
debug: epoch 0   batch 76           16s -> cost 56.389536377263184, accuracy with test points 0.92
debug: epoch 0   batch 77           16s -> cost 55.69497722065817, accuracy with test points 0.92
debug: epoch 0   batch 78           16s -> cost 54.90667918568317, accuracy with test points 0.93
debug: epoch 0   batch 79           16s -> cost 54.07132501779359, accuracy with test points 0.94
debug: epoch 0   batch 80           16s -> cost 53.324068049268156, accuracy with test points 0.94
debug: epoch 0   batch 81           17s -> cost 54.52015109304102, accuracy with test points 0.93
debug: epoch 0   batch 82           17s -> cost 57.82170368101984, accuracy with test points 0.92
debug: epoch 0   batch 83           17s -> cost 59.65782064537309, accuracy with test points 0.92
debug: epoch 0   batch 84           17s -> cost 60.259830510796554, accuracy with test points 0.91
debug: epoch 0   batch 85           17s -> cost 58.994979898102656, accuracy with test points 0.91
debug: epoch 0   batch 86           18s -> cost 57.34351347306775, accuracy with test points 0.92
debug: epoch 0   batch 87           18s -> cost 56.15740393669267, accuracy with test points 0.93
debug: epoch 0   batch 88           18s -> cost 56.043718825661976, accuracy with test points 0.92
debug: epoch 0   batch 89           18s -> cost 57.452356872426314, accuracy with test points 0.93
debug: epoch 0   batch 90           18s -> cost 59.955379827389535, accuracy with test points 0.92
debug: epoch 0   batch 91           18s -> cost 62.8041446584197, accuracy with test points 0.91
debug: epoch 0   batch 92           19s -> cost 65.20561632088521, accuracy with test points 0.9
debug: epoch 0   batch 93           19s -> cost 65.47197313654384, accuracy with test points 0.9
debug: epoch 0   batch 94           19s -> cost 63.96542562603769, accuracy with test points 0.9
debug: epoch 0   batch 95           19s -> cost 61.057823404098045, accuracy with test points 0.9
debug: epoch 0   batch 96           19s -> cost 57.54444784927794, accuracy with test points 0.9
debug: epoch 0   batch 97           19s -> cost 55.799905930751564, accuracy with test points 0.91
debug: epoch 0   batch 98           20s -> cost 55.81691782076183, accuracy with test points 0.91
debug: epoch 0   batch 99           20s -> cost 57.76190157920227, accuracy with test points 0.91
debug: epoch 0   batch 100          20s -> cost 60.09789380621815, accuracy with test points 0.91
debug: epoch 0   batch 101          20s -> cost 62.7385831110778, accuracy with test points 0.93
debug: epoch 0   batch 102          20s -> cost 57.245804410426835, accuracy with test points 0.93
debug: epoch 0   batch 103          20s -> cost 56.039840228697734, accuracy with test points 0.93
debug: epoch 0   batch 104          21s -> cost 57.35768970837701, accuracy with test points 0.92
debug: epoch 0   batch 105          21s -> cost 59.39931113498459, accuracy with test points 0.92
debug: epoch 0   batch 106          21s -> cost 61.54306085452002, accuracy with test points 0.89
debug: epoch 0   batch 107          21s -> cost 62.89548199534188, accuracy with test points 0.91
debug: epoch 0   batch 108          21s -> cost 63.54199862102026, accuracy with test points 0.93
debug: epoch 0   batch 109          21s -> cost 62.42344915015181, accuracy with test points 0.93
debug: epoch 0   batch 110          22s -> cost 60.6699670573182, accuracy with test points 0.94
debug: epoch 0   batch 111          22s -> cost 59.044270612727196, accuracy with test points 0.94
debug: epoch 0   batch 112          22s -> cost 57.959534410097056, accuracy with test points 0.94
debug: epoch 0   batch 113          22s -> cost 57.908753065551764, accuracy with test points 0.95
debug: epoch 0   batch 114          22s -> cost 59.028619302479264, accuracy with test points 0.95
debug: epoch 0   batch 115          23s -> cost 60.64345994573608, accuracy with test points 0.95
debug: epoch 0   batch 116          23s -> cost 61.95188874809762, accuracy with test points 0.91
debug: epoch 0   batch 117          23s -> cost 64.81263733592043, accuracy with test points 0.88
debug: epoch 0   batch 118          23s -> cost 68.57616458581128, accuracy with test points 0.88
debug: epoch 0   batch 119          23s -> cost 71.39903629068728, accuracy with test points 0.88
debug: epoch 0   batch 120          23s -> cost 72.36006328058507, accuracy with test points 0.88
debug: epoch 0   batch 121          24s -> cost 72.9772108434975, accuracy with test points 0.88
debug: epoch 0   batch 122          24s -> cost 70.2327703722788, accuracy with test points 0.9
debug: epoch 0   batch 123          24s -> cost 64.81463706895627, accuracy with test points 0.9
debug: epoch 0   batch 124          24s -> cost 58.554254053993525, accuracy with test points 0.91
debug: epoch 0   batch 125          24s -> cost 53.412179552102764, accuracy with test points 0.93
debug: epoch 0   batch 126          24s -> cost 50.90046617650837, accuracy with test points 0.94
debug: epoch 0   batch 127          25s -> cost 51.48008593714471, accuracy with test points 0.94
debug: epoch 0   batch 128          25s -> cost 53.77639052396825, accuracy with test points 0.92
debug: epoch 0   batch 129          25s -> cost 56.827630905490686, accuracy with test points 0.92
debug: epoch 0   batch 130          25s -> cost 59.442174576669935, accuracy with test points 0.91
debug: epoch 0   batch 131          25s -> cost 59.65356618062218, accuracy with test points 0.91
debug: epoch 0   batch 132          25s -> cost 58.23538915383096, accuracy with test points 0.91
debug: epoch 0   batch 133          26s -> cost 57.29749767081262, accuracy with test points 0.92
debug: epoch 0   batch 134          26s -> cost 57.0837603624543, accuracy with test points 0.93
debug: epoch 0   batch 135          26s -> cost 56.04428797920774, accuracy with test points 0.93
debug: epoch 0   batch 136          26s -> cost 55.53120888494813, accuracy with test points 0.93
debug: epoch 0   batch 137          26s -> cost 54.902804819660695, accuracy with test points 0.94
debug: epoch 0   batch 138          26s -> cost 54.78979240611261, accuracy with test points 0.94
debug: epoch 0   batch 139          27s -> cost 54.84798555958865, accuracy with test points 0.93
debug: epoch 0   batch 140          27s -> cost 54.68157263217209, accuracy with test points 0.92
debug: epoch 0   batch 141          27s -> cost 54.900840857002514, accuracy with test points 0.92
debug: epoch 0   batch 142          27s -> cost 55.00049892952235, accuracy with test points 0.93
debug: epoch 0   batch 143          27s -> cost 55.1259537710335, accuracy with test points 0.92
debug: epoch 0   batch 144          28s -> cost 54.79280552619344, accuracy with test points 0.93
debug: epoch 0   batch 145          28s -> cost 54.42738057615533, accuracy with test points 0.93
debug: epoch 0   batch 146          28s -> cost 53.183010833940706, accuracy with test points 0.92
debug: epoch 0   batch 147          28s -> cost 53.14895348978496, accuracy with test points 0.93
debug: epoch 0   batch 148          28s -> cost 54.083202133586546, accuracy with test points 0.93
debug: epoch 0   batch 149          28s -> cost 55.57371962067966, accuracy with test points 0.92
debug: epoch 0   batch 150          29s -> cost 55.63162536319721, accuracy with test points 0.91
debug: epoch 0   batch 151          29s -> cost 54.54305279663274, accuracy with test points 0.94
debug: epoch 0   batch 152          29s -> cost 51.813481225742194, accuracy with test points 0.94
debug: epoch 0   batch 153          29s -> cost 48.939152716821596, accuracy with test points 0.95
debug: epoch 0   batch 154          29s -> cost 46.928265531760495, accuracy with test points 0.95
debug: epoch 0   batch 155          29s -> cost 45.82452747354323, accuracy with test points 0.95
debug: epoch 0   batch 156          30s -> cost 45.23797387131533, accuracy with test points 0.95
debug: epoch 0   batch 157          30s -> cost 44.81961109773003, accuracy with test points 0.95
debug: epoch 0   batch 158          30s -> cost 44.851105776654094, accuracy with test points 0.95
debug: epoch 0   batch 159          30s -> cost 44.64173352517958, accuracy with test points 0.94
debug: epoch 0   batch 160          30s -> cost 44.42105232472584, accuracy with test points 0.95
debug: epoch 0   batch 161          30s -> cost 44.64701912771521, accuracy with test points 0.95
debug: epoch 0   batch 162          31s -> cost 44.90218418489315, accuracy with test points 0.93
debug: epoch 0   batch 163          31s -> cost 44.38900479858533, accuracy with test points 0.95
debug: epoch 0   batch 164          31s -> cost 44.10211135184916, accuracy with test points 0.95
debug: epoch 0   batch 165          31s -> cost 44.088333809658735, accuracy with test points 0.95
debug: epoch 0   batch 166          31s -> cost 43.42360152074774, accuracy with test points 0.95
debug: epoch 0   batch 167          32s -> cost 41.98438385703422, accuracy with test points 0.95
debug: epoch 0   batch 168          32s -> cost 40.92590109496707, accuracy with test points 0.96
debug: epoch 0   batch 169          32s -> cost 39.962096694503884, accuracy with test points 0.96
debug: epoch 0   batch 170          32s -> cost 40.13355229280598, accuracy with test points 0.95
debug: epoch 0   batch 171          32s -> cost 40.796602666253726, accuracy with test points 0.95
debug: epoch 0   batch 172          32s -> cost 41.59375689896648, accuracy with test points 0.95
debug: epoch 0   batch 173          33s -> cost 42.47005605230528, accuracy with test points 0.95
debug: epoch 0   batch 174          33s -> cost 42.8769875867446, accuracy with test points 0.96
debug: epoch 0   batch 175          33s -> cost 42.787379392650614, accuracy with test points 0.94
debug: epoch 0   batch 176          33s -> cost 42.80825237549408, accuracy with test points 0.94
debug: epoch 0   batch 177          33s -> cost 42.68524824823348, accuracy with test points 0.94
debug: epoch 0   batch 178          33s -> cost 42.167706830378364, accuracy with test points 0.94
debug: epoch 0   batch 179          34s -> cost 41.97137161457328, accuracy with test points 0.94
debug: epoch 0   batch 180          34s -> cost 42.0577175702251, accuracy with test points 0.94