-
Notifications
You must be signed in to change notification settings - Fork 280
Neurons
Neurons are the basic unit of the neural network. They can be connected together, or used to gate connections between other neurons. A Neuron can perform basically 4 operations: project connections, gate connections, activate and propagate.
A neuron can project a connection to another neuron, layer or network (i.e. connect neuron A with network B). Here is how it's done:
var A = new Neuron();
var B = new Architect.Perceptron(2,4,2);
A.project(B); // A now projects a connection to B
Neurons can also self-connect:
A.project(A);
The method project returns a Connection
object, that can be gated by another neuron.
A neuron can gate a connection between two neurons, or a neuron's self-connection. This allows you to create second order neural network architectures.
var A = new Neuron();
var B = new Neuron();
var connection = A.project(B);
var C = new Neuron();
C.gate(connection); // now C gates the connection between A and B
When a neuron activates, it computes its state from all its input connections and squashes it using its activation function, and returns the output (activation). You can provide the activation as a parameter (useful for neurons in the input layer. it has to be a float between 0 and 1). For example:
var A = new Neuron();
var B = new Neuron();
A.project(B);
A.activate(0.5); // 0.5
B.activate(); // 0.3244554645
After an activation, you can teach the neuron what should have been the correct output (a.k.a. train). This is done by backpropagating the error. To use the propagate method you have to provide a learning rate, and a target value (float between 0 and 1).
For example, this is how you can train neuron B to activate 0 when neuron A activates 1:
var A = new Neuron();
var B = new Neuron();
A.project(B);
var learningRate = .3;
for(var i = 0; i < 20000; i++)
{
// when A activates 1
A.activate(1);
// train B to activate 0
B.activate();
B.propagate(learningRate, 0);
}
// test it
A.activate(1);
B.activate(); // 0.006540565760853365
Disconnects the neuron completely, or from another node. If the neuron is in a network, the network might break. Currently working on a connect()
function which will reconnect the neuron in a desired way.
var A = new Neuron();
var B = new Neuron();
A.project(B); // A and B now form a connection
A.disconnect(B); // A and B are disconnected
If you want to disconnect a neuron completely (this is one-sided):
var A = new Neuron();
var B = new Neuron();
var C = new Neuron();
A.project(B); // A projects to B
A.project(C); // A projects to C
A.disconnect(); // all connections are removed
By default, a neuron uses a Logistic Sigmoid as its squashing/activation function, and a random bias. You can change those properties the following way:
var A = new Neuron();
A.squash = Neuron.squash.TANH;
A.bias = 1;
There are 5 built-in squashing functions, but you can also create your own:
- Neuron.squash.LOGISTIC
- Neuron.squash.TANH
- Neuron.squash.IDENTITY
- Neuron.squash.HLIM
- Neuron.squash.ReLU
See more squashing functions here