Stanford Lecture 5 - Convolutional Neural Networks

Brief History

Mark I Perceptron

Widrow and Hoff 1960: Adaline and Madaline

Rumelhart 1986 - first time backpropagation becomes popular

Hinton and Salakhutdinov 2006 - reinvigorated research in deep learning restricted Boltzmann machine

Acoustic Modeling using Deep Belief Networks 2010 Mohamed, Dahl, Hinton

Context-Dependent Pre-trained Deep Neural Networks for Large Vocabulary Speech Recognition 2012 Dahl, Yu, Deng and Acero

Imagenet classification with deep convolutional neural networks 2012 Krizhevsky, Sutskever and Hinton 2012.

What gave rise to CNN?

Hubel and Wiesel: how neurons in the visual cortex works? They put electrodes in the cat’s brain and gave cat different brain stimulus. They measure the response of the neurons to these stimuli.

What they found out is that there are these topographical mapping in the cortex. Nearby cells in the cortex represent nearby regions in the visual field.

1980 Neurocognitron [Fukushima]

Gradient-based learning applied to document recognition [LeCun, etc 1998]

AlexNet 2012

Fast-forward to today: self driving cars, NVIDIA Tesla.

Taigman et al. 2014. Face Recognition

Simonyan et al. 2014. Video classification

Toshev, Szegedy 2014 Pose recognition

Reinforcement learning - Atari Games.

Interpretation and Diagnosis of medical images [Levy et al. 2016]

Classification of galaxies [Dieleman et al. 2014]

Street sign recognition [Sermanet et al. 2011], [Ciresan et al.]

Whale recognition, Kaggle recognition

Aerial maps, Mnih and Hinton 2010

Image Captioning [Vinyals et al., 2015], [Karpathy and Li, 2015]

Convolution Layer

Usually, we created one single vector to store the input data. But in the convolution layer, we preserve the structure of the data. Then we apply a filter to it.

Inline-style: