Multi-layer neural networks

As we have mentioned many times, 1-layer neural nets can only classify linearly separable classes. But there is nothing that prevents us from introducing more layers between the input and the output. These extra layers are called hidden layers. The following diagram demonstrates a 3-layer fully connected neural network with two hidden layers. The input layer has k input neurons, the first hidden layer has n hidden neurons, and the second hidden layer has m hidden neurons. The output, in this example, is the two classes y₁ and y₂. On top is the always-on bias neuron. A unit from one-layer is connected to all units from the previous and following layers (hence fully connected). Each connection has its own weight, w, that is not depicted for reasons of simplicity:

But we are not limited to networks with sequential layers, as shown in the preceding diagram. The neurons and their connections form directed cyclic graphs. In such a graph, the information cannot pass twice from the same neuron (no loops) and it flows in only one direction, from the input to the output. We also chose to organize them in layers; therefore, the layers are also organized in the directed cyclic graph. The network in the preceding diagram is just a special case of a graph whose layers are connected sequentially. The following diagram also depicts a valid neural network with two input layers, two output layers, and randomly interconnected hidden layers. For the sake of simplicity, we've depicted the multiple weights, w, connecting the layers as a single line:

In this section, we introduced the most basic type of neural network, that is, the neuron, and we gradually expanded it to a graph of neurons, organized in layers. But we can think of it in another way. Thus, we came to know that the neuron has a precise mathematical definition. Therefore, the neural network, as a composition of neurons, is also a mathematical function where the input data represents the function arguments and the network weights, w, are its parameters.