Neural Networks in Neuroscience and Computer Science - Revision history

imported>Psych202: /* Sample Application */

2013-06-07T00:34:50Z

Sample Application

← Older revision		Revision as of 00:34, 7 June 2013
Line 85:		Line 85:

	===Sample Application===		===Sample Application===
	One example of neural networks being applied in a real-world setting is training a pedestrian detector. In Andrew Ng's lab, a pedestrian detector neural network was trained using some help from Yann Lecun's lab and the EBLearn framework (http://eblearn.cs.nyu.edu:21991/doku.php). It was then connected to a haptic belt worn across the waist in order to notify a blind user as to where people are around him/her. You can watch the video of Justin Chen wearing the apparatus here: [http://www.youtube.com/watch?v=mQWzaOaSqk8\| Stanford AI Lab Pedestrian Detection Haptic Belt].		One example of neural networks being applied in a real-world setting is training a pedestrian detector. In Andrew Ng's lab, a pedestrian detector neural network was trained using some help from Yann Lecun's lab and the EBLearn framework (http://eblearn.cs.nyu.edu:21991/doku.php). It was then connected to a haptic belt worn across the waist in order to notify a blind user as to where people are around him/her. You can watch the video of Justin Chen wearing the apparatus here: [http://www.youtube.com/watch?v=mQWzaOaSqk8 Stanford AI Lab Pedestrian Detection Haptic Belt].

	=Conclusion=		=Conclusion=

imported>Psych202: /* Sample Application */

2013-06-07T00:34:19Z

Sample Application

← Older revision		Revision as of 00:34, 7 June 2013
Line 85:		Line 85:

	===Sample Application===		===Sample Application===
	One example of neural networks being applied in a real-world setting is training a pedestrian detector. In Andrew Ng's lab, a pedestrian detector neural network was trained using some help from Yann Lecun's lab and the EBLearn framework (http://eblearn.cs.nyu.edu:21991/doku.php). It was then connected to a haptic belt worn across the waist in order to notify a blind user as to where people are around him/her. You can watch the video of Justin Chen wearing the apparatus here: [[http://www.youtube.com/watch?v=mQWzaOaSqk8 \| Stanford AI Lab Pedestrian Detection Haptic Belt]].		One example of neural networks being applied in a real-world setting is training a pedestrian detector. In Andrew Ng's lab, a pedestrian detector neural network was trained using some help from Yann Lecun's lab and the EBLearn framework (http://eblearn.cs.nyu.edu:21991/doku.php). It was then connected to a haptic belt worn across the waist in order to notify a blind user as to where people are around him/her. You can watch the video of Justin Chen wearing the apparatus here: [http://www.youtube.com/watch?v=mQWzaOaSqk8\| Stanford AI Lab Pedestrian Detection Haptic Belt].

	=Conclusion=		=Conclusion=

imported>Psych202: /* Sample Application */

2013-06-07T00:33:17Z

Sample Application

← Older revision		Revision as of 00:33, 7 June 2013
Line 85:		Line 85:

	===Sample Application===		===Sample Application===
	One example of neural networks being applied in a real-world setting is training a pedestrian detector. In Andrew Ng's lab, a pedestrian detector neural network was trained using some help from Yann Lecun's lab and the EBLearn framework (http://eblearn.cs.nyu.edu:21991/doku.php). It was then connected to a haptic belt worn across the waist in order to notify a blind user as to where people are around him/her. You can watch the video of Justin Chen wearing the apparatus here: [[~~Stanford AI Lab Pedestrian Detection Haptic Belt \|~~ http://www.youtube.com/watch?v=mQWzaOaSqk8]].		One example of neural networks being applied in a real-world setting is training a pedestrian detector. In Andrew Ng's lab, a pedestrian detector neural network was trained using some help from Yann Lecun's lab and the EBLearn framework (http://eblearn.cs.nyu.edu:21991/doku.php). It was then connected to a haptic belt worn across the waist in order to notify a blind user as to where people are around him/her. You can watch the video of Justin Chen wearing the apparatus here: [[http://www.youtube.com/watch?v=mQWzaOaSqk8 \| Stanford AI Lab Pedestrian Detection Haptic Belt]].

	=Conclusion=		=Conclusion=

imported>Psych202: /* Neural Networks in Computer Science (Artificial Intelligence) */

2013-06-07T00:32:13Z

Neural Networks in Computer Science (Artificial Intelligence)

← Older revision		Revision as of 00:32, 7 June 2013
Line 83:		Line 83:

	When using a sparse autoencoder trained on 100 hidden units on 10x10 pixel inputs, we can get a lot of features that look like edges at different positions and orientations. When passing a new image through this neural network, edges that are similar to these features will set off the activations and send off "synapses" similar to the biological network. If enough activations are sent out, then the network would recognize the image as positive for the object of interest (such as face, numerical digit, etc.).		When using a sparse autoencoder trained on 100 hidden units on 10x10 pixel inputs, we can get a lot of features that look like edges at different positions and orientations. When passing a new image through this neural network, edges that are similar to these features will set off the activations and send off "synapses" similar to the biological network. If enough activations are sent out, then the network would recognize the image as positive for the object of interest (such as face, numerical digit, etc.).

			===Sample Application===
			One example of neural networks being applied in a real-world setting is training a pedestrian detector. In Andrew Ng's lab, a pedestrian detector neural network was trained using some help from Yann Lecun's lab and the EBLearn framework (http://eblearn.cs.nyu.edu:21991/doku.php). It was then connected to a haptic belt worn across the waist in order to notify a blind user as to where people are around him/her. You can watch the video of Justin Chen wearing the apparatus here: [[Stanford AI Lab Pedestrian Detection Haptic Belt \| http://www.youtube.com/watch?v=mQWzaOaSqk8]].

	=Conclusion=		=Conclusion=

imported>Psych202 at 17:33, 6 June 2013

2013-06-06T17:33:50Z

← Older revision		Revision as of 17:33, 6 June 2013
Line 89:		Line 89:
	Many different neuroimaging techniques have been developed to further understand neural networks from a biological standpoint as well in order to potentially better simulate it artificially. Such brain-scanning technologies include fMRI (functional magnetic resonance imaging), PET (positron emission tomography) and CAT (computed axial tomography). Lesioning studies can also yield important insights on the working of several cell assemblies (e.g. removing nodes in an artificial neural network to see what the effect is).		Many different neuroimaging techniques have been developed to further understand neural networks from a biological standpoint as well in order to potentially better simulate it artificially. Such brain-scanning technologies include fMRI (functional magnetic resonance imaging), PET (positron emission tomography) and CAT (computed axial tomography). Lesioning studies can also yield important insights on the working of several cell assemblies (e.g. removing nodes in an artificial neural network to see what the effect is).

	It is still unclear how well ANN's can model the brain's processes, but as more powerful and complex neural networks (convolutional neural network, deep belief networks, recursive neural networks) are being developed, we are getting closer to recreating the immense computational power that the brain has.		It is still unclear how well ANN's can model the brain's processes, but as more powerful and complex neural networks (convolutional neural network, deep belief networks<sup>[[Neural_Networks_in_Neuroscience_and_Computer_Science#References\|[7]]]</sup>, recursive neural networks) are being developed, we are getting closer to recreating the immense computational power that the brain has.

	=References=		=References=
Line 98:		Line 98:
	# Wang T, Wu D, Coates A, Ng A. End-to-End Text Recognition with Convolutional Neural Networks. Retrieved from: http://ai.stanford.edu/~ang/papers/ICPR12-TextRecognitionConvNeuralNets.pdf.		# Wang T, Wu D, Coates A, Ng A. End-to-End Text Recognition with Convolutional Neural Networks. Retrieved from: http://ai.stanford.edu/~ang/papers/ICPR12-TextRecognitionConvNeuralNets.pdf.
	# NASA Neural Network Project Passes Milestone. 2003. Retrieved from: http://www.nasa.gov/centers/dryden/news/NewsReleases/2003/03-49.html.		# NASA Neural Network Project Passes Milestone. 2003. Retrieved from: http://www.nasa.gov/centers/dryden/news/NewsReleases/2003/03-49.html.
			# Deep Networks: Overview. Retrieved from: http://ufldl.stanford.edu/wiki/index.php/Deep_Networks:_Overview.

imported>Psych202: /* Conclusion */

2013-06-06T17:33:12Z

Conclusion

← Older revision		Revision as of 17:33, 6 June 2013
Line 88:		Line 88:

	Many different neuroimaging techniques have been developed to further understand neural networks from a biological standpoint as well in order to potentially better simulate it artificially. Such brain-scanning technologies include fMRI (functional magnetic resonance imaging), PET (positron emission tomography) and CAT (computed axial tomography). Lesioning studies can also yield important insights on the working of several cell assemblies (e.g. removing nodes in an artificial neural network to see what the effect is).		Many different neuroimaging techniques have been developed to further understand neural networks from a biological standpoint as well in order to potentially better simulate it artificially. Such brain-scanning technologies include fMRI (functional magnetic resonance imaging), PET (positron emission tomography) and CAT (computed axial tomography). Lesioning studies can also yield important insights on the working of several cell assemblies (e.g. removing nodes in an artificial neural network to see what the effect is).

			It is still unclear how well ANN's can model the brain's processes, but as more powerful and complex neural networks (convolutional neural network, deep belief networks, recursive neural networks) are being developed, we are getting closer to recreating the immense computational power that the brain has.

	=References=		=References=

imported>Psych202: /* Supervised Learning */

2013-06-06T17:29:21Z

Supervised Learning

← Older revision		Revision as of 17:29, 6 June 2013
Line 41:		Line 41:
	J(W,b; x,y) = \frac{1}{2} \left\\| h_{W,b}(x) - y \right\\|^2.		J(W,b; x,y) = \frac{1}{2} \left\\| h_{W,b}(x) - y \right\\|^2.
	</math>		</math>
	This is a (one-half) squared-error cost function. Given a training set of <math>m</math> examples, ~~we then define~~ the overall cost function ~~to be~~:		This is a (one-half) squared-error cost function. Given a training set of <math>m</math> examples, the overall cost function is defined as:
	:<math>		:<math>
	J(W,b) = [ \frac{1}{m} \sum_{i=1}^m J(W,b;x^{(i)},y^{(i)}) ]		J(W,b) = [ \frac{1}{m} \sum_{i=1}^m J(W,b;x^{(i)},y^{(i)}) ]
Line 55:		Line 55:
	The first term in the definition of <math>J(W,b)</math> is an average sum-of-squares error term. The second term is a regularization term (also called a '''weight decay''' term) that tends to decrease the magnitude of the weights, and helps prevent overfitting.		The first term in the definition of <math>J(W,b)</math> is an average sum-of-squares error term. The second term is a regularization term (also called a '''weight decay''' term) that tends to decrease the magnitude of the weights, and helps prevent overfitting.

	The '''weight decay parameter''' <math>\lambda</math> controls the relative importance of the two terms. ~~Note also the slightly overloaded notation:~~ <math>J(W,b;x,y)</math> is the squared error cost with respect to a single example; <math>J(W,b)</math> is the overall cost function, which includes the weight decay term.		The '''weight decay parameter''' <math>\lambda</math> controls the relative importance of the two terms. <math>J(W,b;x,y)</math> is the squared error cost with respect to a single example whereas <math>J(W,b)</math> is the overall cost function, which includes the weight decay term.

	Our goal is to minimize <math>J(W,b)</math> as a function of <math>W</math> and <math>b</math>. To train our neural network, we initialize all the network parameters to a small random value near zero and then apply an optimization algorithm such as batch gradient descent. Since <math>J(W,b)</math> is a a non-convex function, gradient descent could converge to a local optima, however, in practice, it works fairly well.		Our goal is to minimize <math>J(W,b)</math> as a function of <math>W</math> and <math>b</math>. To train our neural network, we initialize all the network parameters to a small random value near zero and then apply an optimization algorithm such as batch gradient descent. Since <math>J(W,b)</math> is a a non-convex function, gradient descent could converge to a local optima, however, in practice, it works fairly well.

imported>Psych202: /* Introduction */

2013-06-06T17:27:39Z

Introduction

← Older revision		Revision as of 17:27, 6 June 2013
Line 4:		Line 4:

	=Introduction=		=Introduction=
	A '''Neural Network''' is a network of neurons working together to send a flow of signals to accomplish some task. The original biological neural networks consist of neurons which interact with their neighbors through axon terminals connected via synapses to dendrites in other neurons. A neural circuit is a functional entity of interconnected neurons that regulates its own activity using a feedback loop. Artificial intelligence in the field of Computer Science adopted this information processing paradigm to create artificial neural networks. These artificial neural networks have been applied successfully to speech recognition, image analysis, and recognition tasks. Lots of research in Professor Andrew Ng's lab is geared towards applying neural networks to unsupervised learning tasks. <sup>[[Neural_Networks_in_Neuroscience_and_Computer_Science#References\|[1]]]</sup>		A '''Neural Network''' is a network of neurons working together to send a flow of signals to accomplish some task. The original biological neural networks consist of neurons which interact with their neighbors through axon terminals connected via synapses to dendrites in other neurons. A neural circuit is a functional entity of interconnected neurons that regulates its own activity using a feedback loop. Artificial intelligence in the field of Computer Science adopted this information processing paradigm to create artificial neural networks. These artificial neural networks have been applied successfully to speech recognition, image analysis, and recognition tasks. Lots of research in Stanford's own Professor Andrew Ng's lab is geared towards applying neural networks to unsupervised learning tasks. <sup>[[Neural_Networks_in_Neuroscience_and_Computer_Science#References\|[1]]]</sup>

	=Neural Networks in Neuroscience=		=Neural Networks in Neuroscience=

imported>Psych202: /* Supervised Learning */

2013-06-06T17:26:52Z

Supervised Learning

← Older revision		Revision as of 17:26, 6 June 2013
Line 69:		Line 69:
	</math>		</math>

	The key component is the gradients which can be calculated using a back-propagation algorithm. The intuition behind the back-propagation algorithm is as follows. Given a training example <math>(x,y)</math>, we will first run a "forward pass" to compute all the activations throughout the network, including the output value of the hypothesis <math>h_{W,b}(x)</math>. Then, for each node <math>i</math> in layer <math>l</math>, we would like to compute an "error term" <math>\delta^{(l)}_i</math> that measures how much that node was "responsible" for any errors in our output. For an output node, we can directly measure the difference between the network's activation and the true target value, and use that to define <math>\delta^{(n_l)}_i</math> (where layer <math>n_l</math> is the output layer). For the hidden units, we compute <math>\delta^{(l)}_i</math> based on a weighted average of the error terms of the nodes that uses <math>a^{(l)}_i</math> as an input.		The key component is the gradients which can be calculated using a '''back-propagation algorithm'''. The intuition behind the back-propagation algorithm is as follows. Given a training example <math>(x,y)</math>, we will first run a "forward pass" to compute all the activations throughout the network, including the output value of the hypothesis <math>h_{W,b}(x)</math>. Then, for each node <math>i</math> in layer <math>l</math>, we would like to compute an "error term" <math>\delta^{(l)}_i</math> that measures how much that node was "responsible" for any errors in our output. For an output node, we can directly measure the difference between the network's activation and the true target value, and use that to define <math>\delta^{(n_l)}_i</math> (where layer <math>n_l</math> is the output layer). For the hidden units, we compute <math>\delta^{(l)}_i</math> based on a weighted average of the error terms of the nodes that uses <math>a^{(l)}_i</math> as an input.

	To train our neural network, we can now repeatedly take steps of gradient descent to reduce our cost function <math>J(W,b)</math>		To train our neural network, we can now repeatedly take steps of gradient descent to reduce our cost function <math>J(W,b)</math>

imported>Psych202: /* Receptive fields */

2013-06-06T17:26:22Z

Receptive fields

← Older revision		Revision as of 17:26, 6 June 2013
Line 18:		Line 18:

	===Receptive fields===		===Receptive fields===
	A receptive field is a small region within an entire visual field and is commonly used in the ideas of convolutional neural networks. Any given neuron only responds to a subset of stimuli within its receptive field (tuning). A neuron in V1 may fire to any vertical stimulus in its receptive field because of its simple tuning, but in higher visual areas such as the fusiform gyrus, the neuron may only fire when a certain face appears in the receptive field. Memories are very likely represented by patterns of activation amongst these neural networks. The study and modeling of these networks has attracted a lot of interest in many different fields. It has the potential to explain various aspects of behavior, learning, and memory. The most important property of neural networks is the ability to learn complex patterns which is heavily emphasized in other fields like Computer Science.		A '''receptive field''' is a small region within an entire visual field and is commonly used in the ideas of '''convolutional neural networks'''. Any given neuron only responds to a subset of stimuli within its receptive field (tuning). A neuron in V1 may fire to any vertical stimulus in its receptive field because of its simple tuning, but in higher visual areas such as the fusiform gyrus, the neuron may only fire when a certain face appears in the receptive field. Memories are very likely represented by patterns of activation amongst these neural networks. The study and modeling of these networks has attracted a lot of interest in many different fields. It has the potential to explain various aspects of behavior, learning, and memory. The most important property of neural networks is the ability to learn complex patterns which is heavily emphasized in other fields like Computer Science.

	=Neural Networks in Computer Science (Artificial Intelligence)=		=Neural Networks in Computer Science (Artificial Intelligence)=