An Introduction to Convolutional Neural Networks - Revision history

imported>Psych202: /* Feature Universality */

2013-06-09T00:03:09Z

Feature Universality

← Older revision		Revision as of 00:03, 9 June 2013
Line 39:		Line 39:
	==== The Issue of Feedback ====		==== The Issue of Feedback ====
	All of the current convolutional networks share the common problem of being strictly feed-forward. While its unclear if this limits their performance as machine learning tools, there is clear evidence from neuroimaging that feed-back connections do exist in the relevant cortical structures.		All of the current convolutional networks share the common problem of being strictly feed-forward. While its unclear if this limits their performance as machine learning tools, there is clear evidence from neuroimaging that feed-back connections do exist in the relevant cortical structures.
	==== Feature Universality ====		==== Feature Depth & Universality ====
			While current model can be trained up to around 8 functional layers, many more might be needed to scale up to the full complexity of the human ventral stream. Unfortunately, the prospects for such scaling seem bleak, since backpropagation is known to be subject to "disappearing gradiants", error signals that rapidly approach zero as there passed down the layers. Luckily, there are several ways in which researchers are fighting back against this problem. The first comes from neuroscience, where some (e.g. Serre), have mapped their architectures directly to neural structure, and the mapping suggests that maybe to only really have 8 or so layers. Another approach is to evoke evolution. Perhaps some convolution layers were learnt on an evolutionary timescale and are task invariant. Such universal features aren't all that implausible for the early layers, as things such as Gabor filters tend to be learnt across most task settings. The last, and most promising, approach is to look to the emerging "deep learning" community for insight. So called deep architectures can have arbitrarily many layers since they are trained in a greedy, layer-by-layer procedure. While the best greedy learning algorithm for convolutional architectures is currently unclear (most deep learning involves unsupervised error signals), inroads are being made and the future for convolutional networks remains bright.

	== References ==		== References ==

imported>Psych202: /* Open Questions */

2013-06-08T23:53:34Z

Open Questions

← Older revision		Revision as of 23:53, 8 June 2013
Line 36:		Line 36:
	=== Open Questions ===		=== Open Questions ===
	==== Differing Implementations ====		==== Differing Implementations ====
	While the LeNet is impressive, others have proposed slightly different architectures that can account for different psychological data and outperform it in a computer vision context <sup>[[An_Introduction_to_Convolutional_Neural_Networks#References\|[4]]]</sup>.		While the LeNet is impressive, others have proposed slightly different architectures that can account for different psychological data and outperform it in a computer vision context <sup>[[An_Introduction_to_Convolutional_Neural_Networks#References\|[4]]]</sup>. However, due to the vast number of ways one might tweak such an architecture, its unclear how these various model might be evaluated with respect to their neural validity. More explicit predictions about human behavior and neural anatomy must be made before this has a hope of being resolved.
	==== The Issue of Feedback ====		==== The Issue of Feedback ====
			All of the current convolutional networks share the common problem of being strictly feed-forward. While its unclear if this limits their performance as machine learning tools, there is clear evidence from neuroimaging that feed-back connections do exist in the relevant cortical structures.
	==== Feature Universality ====		==== Feature Universality ====

imported>Psych202 at 23:47, 8 June 2013

2013-06-08T23:47:51Z

imported>Psych202: /* Open Questions */

2013-06-08T23:29:34Z

Open Questions

← Older revision		Revision as of 23:29, 8 June 2013
Line 36:		Line 36:
	=== Open Questions ===		=== Open Questions ===
	==== Differing Implementations ====		==== Differing Implementations ====
			While the LeNet is impressive, others
	==== The Issue of Feedback ====		==== The Issue of Feedback ====
	==== Feature Universality ====		==== Feature Universality ====

imported>Psych202: /* Convolutional Neural Networks */

2013-06-08T23:29:01Z

Convolutional Neural Networks

← Older revision		Revision as of 23:29, 8 June 2013
Line 15:		Line 15:

	== Convolutional Neural Networks ==		== Convolutional Neural Networks ==
	~~[[Image:Convo1d.png\|thumb\|400px\|right\|Color coding which weights are shared between neuronal connections.]]~~
	~~[[Image:SparseConnect.png\|thumb\|400px\|right\|The sparse connectivity responsible for the receptive field behavior.]]~~
	The solution to FFNNs' problems with image processing took inspiration from neurobiology, Yann LeCun and Toshua Bengio tried to capture the organization of neurons in the visual cortex of the cat, which at that time was known to consist of maps of local receptive fields that decreased in granularity as the cortex moved anteriorly. There are several different theory about how to precisely define such a model, but all of the various implementations can be loosely described as involving the following process:		The solution to FFNNs' problems with image processing took inspiration from neurobiology, Yann LeCun and Toshua Bengio tried to capture the organization of neurons in the visual cortex of the cat, which at that time was known to consist of maps of local receptive fields that decreased in granularity as the cortex moved anteriorly. There are several different theory about how to precisely define such a model, but all of the various implementations can be loosely described as involving the following process:
	#Convolve several small filters on the input image		#Convolve several small filters on the input image
Line 25:		Line 23:
	=== The LeCun Formulation ===		=== The LeCun Formulation ===
	There are several different ways one might formalize the high level process described above, but the most common is LeCun's implementation, the LeNet.		There are several different ways one might formalize the high level process described above, but the most common is LeCun's implementation, the LeNet.
			[[Image:Convo1d.png\|thumb\|400px\|right\|Color coding which weights are shared between neuronal connections.]]
			[[Image:SparseConnect.png\|thumb\|400px\|right\|The sparse connectivity responsible for the receptive field behavior.]]
	==== Convolution ====		==== Convolution ====
	Convolution is a mathematical term, defined as applying a function repeatedly across the output of another function. In this context it means to apply a 'filter' over an image at all possible offsets. A filter consists of a layer of connection weights, with the input being the size of a small 2D image patch, and the output being a single unit. Since this filter is applying repeatedly, the resulting connectivity looks like a series of overlapping receptive fields, as shown in the 'sparse connectivity' image, which map to a matrix of the filter outputs (or several such matrices in the common case of using a bank of several filters). An important subtlety here is that why there are still a good deal of connections between the input layer and the filter output layer, the weights are tied together (as shown in the colored diagram). This means that during backpropagation, you only have to adjust a number of parameters equal to a ''single'' instance of the filter -- a drastic reduction from the typical FFNN architecture. Another nuance is that we could sensibly apply such filter to any input that's spatially organized, not just a picture. This means that we could add another bank of filters directly on top of our first filter bank's output. However, since the dimensionality of applying a filter is equal to the input dimensionality, we wouldn't be gaining any translation invariance with these additional filters, we'd be stuck doing pixel-wise analysis on increasingly abstract features. In order to solve this problem, we must introduce a new sort of layer: a subsampling layer.		Convolution is a mathematical term, defined as applying a function repeatedly across the output of another function. In this context it means to apply a 'filter' over an image at all possible offsets. A filter consists of a layer of connection weights, with the input being the size of a small 2D image patch, and the output being a single unit. Since this filter is applying repeatedly, the resulting connectivity looks like a series of overlapping receptive fields, as shown in the 'sparse connectivity' image, which map to a matrix of the filter outputs (or several such matrices in the common case of using a bank of several filters). An important subtlety here is that why there are still a good deal of connections between the input layer and the filter output layer, the weights are tied together (as shown in the colored diagram). This means that during backpropagation, you only have to adjust a number of parameters equal to a ''single'' instance of the filter -- a drastic reduction from the typical FFNN architecture. Another nuance is that we could sensibly apply such filter to any input that's spatially organized, not just a picture. This means that we could add another bank of filters directly on top of our first filter bank's output. However, since the dimensionality of applying a filter is equal to the input dimensionality, we wouldn't be gaining any translation invariance with these additional filters, we'd be stuck doing pixel-wise analysis on increasingly abstract features. In order to solve this problem, we must introduce a new sort of layer: a subsampling layer.

imported>Psych202: /* Putting it all together */

2013-06-08T23:28:19Z

Putting it all together

← Older revision		Revision as of 23:28, 8 June 2013
Line 31:		Line 31:

	==== Putting it all together ====		==== Putting it all together ====
	[[Image:mylenet.png\|thumb\|3200px\|center]]		[[Image:mylenet.png\|thumb\|3200px\|center\|The complete implementation of the LeNet.]]
			Even with the convolution and subsampling layers specified, there are still many free hyper-parameters. Namely, how many filters per convolutional layer? How big should the filters and subsamples be? and how many overall layers should there be? None of these questions be answered definitively, as the effectiveness of each hyper-parameter setting depends on the task setting, but LeCun attempted to provide some plausible values for roughing simulating human performance on natural image classification tasks. As the above figure shows, there are five functional layers in the total architecture, corresponding to 2 sets of convolution-max-pooling pairs and a FFNN for solving the actually classification problem. While training takes quite some time, this network learns much faster than a standard FFNN and performances quite well as a pure piece of computer vision software, which is rather incredible considering it wasn't developed solely for machine learning purposes.

	=== Open Questions ===		=== Open Questions ===

imported>Psych202: /* Putting it all together */

2013-06-08T23:16:04Z

Putting it all together

← Older revision		Revision as of 23:16, 8 June 2013
Line 31:		Line 31:

	==== Putting it all together ====		==== Putting it all together ====
			[[Image:mylenet.png\|thumb\|3200px\|center]]

	=== Open Questions ===		=== Open Questions ===

imported>Psych202: /* Subsampling */

2013-06-08T23:13:45Z

Subsampling

← Older revision		Revision as of 23:13, 8 June 2013
Line 28:		Line 28:
	Convolution is a mathematical term, defined as applying a function repeatedly across the output of another function. In this context it means to apply a 'filter' over an image at all possible offsets. A filter consists of a layer of connection weights, with the input being the size of a small 2D image patch, and the output being a single unit. Since this filter is applying repeatedly, the resulting connectivity looks like a series of overlapping receptive fields, as shown in the 'sparse connectivity' image, which map to a matrix of the filter outputs (or several such matrices in the common case of using a bank of several filters). An important subtlety here is that why there are still a good deal of connections between the input layer and the filter output layer, the weights are tied together (as shown in the colored diagram). This means that during backpropagation, you only have to adjust a number of parameters equal to a ''single'' instance of the filter -- a drastic reduction from the typical FFNN architecture. Another nuance is that we could sensibly apply such filter to any input that's spatially organized, not just a picture. This means that we could add another bank of filters directly on top of our first filter bank's output. However, since the dimensionality of applying a filter is equal to the input dimensionality, we wouldn't be gaining any translation invariance with these additional filters, we'd be stuck doing pixel-wise analysis on increasingly abstract features. In order to solve this problem, we must introduce a new sort of layer: a subsampling layer.		Convolution is a mathematical term, defined as applying a function repeatedly across the output of another function. In this context it means to apply a 'filter' over an image at all possible offsets. A filter consists of a layer of connection weights, with the input being the size of a small 2D image patch, and the output being a single unit. Since this filter is applying repeatedly, the resulting connectivity looks like a series of overlapping receptive fields, as shown in the 'sparse connectivity' image, which map to a matrix of the filter outputs (or several such matrices in the common case of using a bank of several filters). An important subtlety here is that why there are still a good deal of connections between the input layer and the filter output layer, the weights are tied together (as shown in the colored diagram). This means that during backpropagation, you only have to adjust a number of parameters equal to a ''single'' instance of the filter -- a drastic reduction from the typical FFNN architecture. Another nuance is that we could sensibly apply such filter to any input that's spatially organized, not just a picture. This means that we could add another bank of filters directly on top of our first filter bank's output. However, since the dimensionality of applying a filter is equal to the input dimensionality, we wouldn't be gaining any translation invariance with these additional filters, we'd be stuck doing pixel-wise analysis on increasingly abstract features. In order to solve this problem, we must introduce a new sort of layer: a subsampling layer.
	==== Subsampling ====		==== Subsampling ====
			Subsampling, or down-sampling, refers to reducing the overall size of a signal. In many cases, such as audio compression for music files, subsampling is done simply for the size reduction. But in the domain of 2D filter outputs, subsampling can also be thought of as increasing the position invariance of the filters. The specific subsampling method used in LeNets is known as 'max pooling'. This involves splitting up the matrix of filter outputs into small non-overlapping grids (the larger the grid, the greater the signal reduction), and taking the maximum value in each grid as the value in the reduced matrix. Semantically, this corresponds to changing the question answered by the convolution layer from "how well does this filter apply right here" to "how well does this filter apply to this area". Now, by applying such a max pooling layer in between convolutional layers, we can increase spatial abstractness as we increase feature abstractness.

	==== Putting it all together ====		==== Putting it all together ====

imported>Psych202 at 22:59, 8 June 2013

2013-06-08T22:59:17Z

← Older revision		Revision as of 22:59, 8 June 2013
Line 26:		Line 26:
	There are several different ways one might formalize the high level process described above, but the most common is LeCun's implementation, the LeNet.		There are several different ways one might formalize the high level process described above, but the most common is LeCun's implementation, the LeNet.
	==== Convolution ====		==== Convolution ====
	Convolution is a mathematical term, defined as applying a function repeatedly across the output of another function. In this context it means to apply a 'filter' over an image at all possible offsets. A filter consists of a layer of connection weights, with the input being the size of a small 2D image patch, and the output being a single unit. Since this filter is applying repeatedly, the resulting connectivity looks like a series of overlapping receptive fields, which map to a matrix of the filter outputs.		Convolution is a mathematical term, defined as applying a function repeatedly across the output of another function. In this context it means to apply a 'filter' over an image at all possible offsets. A filter consists of a layer of connection weights, with the input being the size of a small 2D image patch, and the output being a single unit. Since this filter is applying repeatedly, the resulting connectivity looks like a series of overlapping receptive fields, as shown in the 'sparse connectivity' image, which map to a matrix of the filter outputs (or several such matrices in the common case of using a bank of several filters). An important subtlety here is that why there are still a good deal of connections between the input layer and the filter output layer, the weights are tied together (as shown in the colored diagram). This means that during backpropagation, you only have to adjust a number of parameters equal to a ''single'' instance of the filter -- a drastic reduction from the typical FFNN architecture. Another nuance is that we could sensibly apply such filter to any input that's spatially organized, not just a picture. This means that we could add another bank of filters directly on top of our first filter bank's output. However, since the dimensionality of applying a filter is equal to the input dimensionality, we wouldn't be gaining any translation invariance with these additional filters, we'd be stuck doing pixel-wise analysis on increasingly abstract features. In order to solve this problem, we must introduce a new sort of layer: a subsampling layer.
	==== Subsampling ====		==== Subsampling ====

imported>Psych202: /* Convolutional Neural Networks */

2013-06-08T22:47:00Z

Convolutional Neural Networks

← Older revision		Revision as of 22:47, 8 June 2013
Line 26:		Line 26:
	There are several different ways one might formalize the high level process described above, but the most common is LeCun's implementation, the LeNet.		There are several different ways one might formalize the high level process described above, but the most common is LeCun's implementation, the LeNet.
	==== Convolution ====		==== Convolution ====
	Convolution is a mathematical term, defined as applying a function repeatedly across the output of another function. In this context it means to apply a 'filter' over an image at all possible offsets. A filter consists of a layer of connection weights, with the input being the size of a small 2D image patch, and the output being a single unit. Since this filter is applying repeatedly, the resulting		Convolution is a mathematical term, defined as applying a function repeatedly across the output of another function. In this context it means to apply a 'filter' over an image at all possible offsets. A filter consists of a layer of connection weights, with the input being the size of a small 2D image patch, and the output being a single unit. Since this filter is applying repeatedly, the resulting connectivity looks like a series of overlapping receptive fields, which map to a matrix of the filter outputs.
	==== Subsampling ====		==== Subsampling ====

← Older revision		Revision as of 23:47, 8 June 2013
Line 2:		Line 2:
	=== The Early Neural Network Models ===		=== The Early Neural Network Models ===
	[[Image:Perceptron.png\|thumb\|300px\|right\|A single layer neural network from Minsky and Papert's book.]]		[[Image:Perceptron.png\|thumb\|300px\|right\|A single layer neural network from Minsky and Papert's book.]]
	Computational models of neural networks have been around for more than half a century, beginning with the simple model that McCulloch and Pitts developed in 1943 <sup>[[An_Introduction_to_Convolutional_Neural_Networks#References\|[1]]]</sup>. Hebb subsequently contributed a learning algorithm to train such models <sup>[[An_Introduction_to_Convolutional_Neural_Networks#References\|[2]]]</sup>, summed up by the familiar refrain: 'Neuron that fire together, wire together'. Hebb's rule, and a popular variant known as the Delta rule, were crucial for early models of cognition, but they quickly ran into trouble with respect to their computational power. In their extremely influencial book, ''Perceptrons'', Minsky and Papert proved that these networks couldn't even learn the boolean XOR function, due to due only being able to learn a single layer of weights<sup>[[An_Introduction_to_Convolutional_Neural_Networks#References\|[3]]]</sup>. Luckily, a more complex learning algorithm, backpropagation, eventually emerged, which could learn across arbitrarily many layers, and was later proven to be capable of approximating any computable function.		Computational models of neural networks have been around for more than half a century, beginning with the simple model that McCulloch and Pitts developed in 1943 <sup>[[An_Introduction_to_Convolutional_Neural_Networks#References\|[1]]]</sup>. Hebb subsequently contributed a learning algorithm to train such models <sup>[[An_Introduction_to_Convolutional_Neural_Networks#References\|[2]]]</sup>, summed up by the familiar refrain: 'Neuron that fire together, wire together'. Hebb's rule, and a popular variant known as the Delta rule, were crucial for early models of cognition, but they quickly ran into trouble with respect to their computational power. In their extremely influencial book, ''Perceptrons'', Minsky and Papert proved that these networks couldn't even learn the boolean XOR function, due to due only being able to learn a single layer of weights<sup>[[An_Introduction_to_Convolutional_Neural_Networks#References\|[3]]]</sup>.Luckily, a more complex learning algorithm, backpropagation, eventually emerged, which could learn across arbitrarily many layers, and was later proven to be capable of approximating any computable function.

	[[Image:feedforward.jpg\|thumb\|300px\|left\|A typical feed-forward neural network architecture used in backpropagation.]]		[[Image:feedforward.jpg\|thumb\|300px\|left\|A typical feed-forward neural network architecture used in backpropagation.]]
Line 36:		Line 36:
	=== Open Questions ===		=== Open Questions ===
	==== Differing Implementations ====		==== Differing Implementations ====
	While the LeNet is impressive, others		While the LeNet is impressive, others have proposed slightly different architectures that can account for different psychological data and outperform it in a computer vision context <sup>[[An_Introduction_to_Convolutional_Neural_Networks#References\|[4]]]</sup>.
	==== The Issue of Feedback ====		==== The Issue of Feedback ====
	==== Feature Universality ====		==== Feature Universality ====
Line 44:		Line 44:
	#Hebb, Donald (1949). The Organization of Behavior. New York: Wiley.		#Hebb, Donald (1949). The Organization of Behavior. New York: Wiley.
	#Minsky, M. and Papert, S. (1969). Perceptrons: An Introduction to Computational Geometry. MIT Press, Cambridge, MA.		#Minsky, M. and Papert, S. (1969). Perceptrons: An Introduction to Computational Geometry. MIT Press, Cambridge, MA.
			#Serre, T., Wolf, L., Bileschi, S., and Riesenhuber, M. (2007). Robust object recog- nition with cortex-like mechanisms. IEEE Trans. Pattern Anal. Mach. Intell., 29(3), 411–426. Member-Poggio, Tomaso.