Why is there only one hidden layer in a neural network?

Question

I recently made my first neural network simulation which also uses a genetic evolution algorithm. It's simple software that just simulates simple organisms collecting food, and they evolve, as one would expect, from organisms with random and sporadic movements into organisms with controlled, food-seeking movements. Since this kind of organism is so simple, I only used a few hidden layer neurons and a few input and output neurons. I understand that more complex neural networks could be made by simply adding more neurons, but can't you add more layers? Or would this create some kind of redundancy? All of the pictures of diagrams of neural networks, such as this one http://mechanicalforex.com/wp-content/uploads/2011/06/NN.png, always have one input layer, one hidden layer, and one output layer. Couldn't a more complex neural network be made if you just added a bunch of hidden layers? Of course this would make processing the neural network harder, but would it create any sort of advantage, or would it be just the same as adding more neurons to a single layer?

Answer 1

You can include as many hidden layers you want, starting from zero (--that case is called perceptron).

The ability to represent unknown functions, however, does -- in principle -- not increase. Single-hidden layer neural networks already possess a universal representation property: by increasing the number of hidden neurons, they can fit (almost) arbitrary functions. You can't get more than this. And particularly not by adding more layers.

However, that doesn't mean that multi-hidden-layer ANN's can't be useful in practice. Yet, as you get another dimension in your parameter set, people usually stuck with the single-hidden-layer version.