too big of a neural net??

[Mcilie]

so i am making an NN to classify mnist digits. Here is the Code:

var sy = require("synaptic");

// create the network
var inputLayer = new sy.Layer(28*28);
var hiddenLayer = new sy.Layer(10);
var outputLayer = new sy.Layer(1);

inputLayer.project(hiddenLayer);
hiddenLayer.project(outputLayer);

var myNetwork = new sy.Network({
    input: inputLayer,
    hidden: [hiddenLayer],
    output: outputLayer
});

// train the network
var learningRate = .3;
for (var i = 0; i < 20; i++)
{
    console.log(i);

    for(var j = 0; j< MyData.length; j++){
        myNetwork.activate(training_set_inputs[j]);
        myNetwork.propagate(learningRate, [training_set_inputs[0][j]]);
    }
}

I have some problems

A) even if i divide all my classes by 10, meaning that they are decimal,
the neural net wont predict anything above 0.003. I know I only trained it 3 times, but its soo slow.
B) if I try to train it more than 784 samples, it starts predicting NaN. 784 is the size of my input layer but I dont see how my dataset size would impact my neural nets ability to train more values.

I dont know what activation function this is, but i think its sigmoid (correct me If im wrong)
Aside from adding more epochs, how can I make my neural net predict 10 classes.
its ok if its in decimal form, ie 0.0 ... 0.9.
thanks

[ghost]

If your NN is too big consider this:

• Use Binary Cost function
• Train only 1 dataset ahead and enable backprop
• Normalize correctly with boundaries far away from 0-1, use 0.0001 to 0.9999
• Use a 8700K for max single thread speed (not sure if V8 supports AVX-512 already ?)

If its still to slow move away from CPU bound NNs to GPU which means Tensorflow with a high end Nvidia card like 1080 and up.

[Mcilie]

okay, but Im wondering why its only predicting like .003, like there isnt even a variation in the predictions. So while this definitley may be too big to nodejs, im pretty sure there is something wrong with the way I train it as well. Also, what activation function does the NN use by default?

[ghost]

Thats right, in my model I have "only" 27 inputs and the error goes super low like 0.0000001. Debug your training sets by printing each step and each single value for like 2-3 training steps. Its a mess to read but required to find bugs.