Chapter 3. Deep Learning Fundamentals

In Chapter 1, Machine Learning – An Introduction, we introduced machine learning and some of its applications, and we briefly talked about a few different algorithms and techniques that can be used to implement machine learning. In Chapter 2, Neural Networks, we concentrated on neural networks; we have shown that 1-layer networks are too simple and can only work on linear problems, and we have introduced the Universal Approximation Theorem, showing how 2-layer neural networks with just one hidden layer are able to approximate to any degree any continuous function on a compact subset of R _n.

In this chapter, we will introduce deep learning and deep neural networks, that is, neural networks with at least two or more hidden layers. The reader may wonder what is the point of using more than one hidden layer, given the Universal Approximation Theorem, and this is in no way a na?ve question, since for a long period the neural networks used were very shallow, with just one hidden layer. The answer is that it is true that 2-layer neural networks can approximate any continuous function to any degree, however, it is also true that adding layers adds levels of complexity that may be much harder and may require many more neurons to simulate with shallow networks. There is also another, more important, reason behind the term deep of deep learning that refers not just to the depth of the network, or how many layers the neural net has, but to the level of "learning". In deep learning, the network does not simply learn to predict an output Y given an input X, but it also understands basic features of the input. In deep learning, the neural network is able to make abstractions of the features that comprise the input examples, to understand the basic characteristics of the examples, and to make predictions based on those characteristics. In deep learning, there is a level of abstraction that is missing in other basic machine learning algorithms or in shallow neural networks.

In this chapter, we will cover the following topics:

• What is deep learning?
• Fundamental concepts of deep learning
• Applications of deep learning
• GPU versus CPU
• Popular open source libraries