Pavlov’s Dog and Artificial Neural Network (Part 1: Perceptron)

Ivan Pavlov’s experiment on Classical Conditioning is one of my favorites in natural science. Part of the reason is that the experiment was conducted with dogs (I’m a dog person). But most importantly, his research has provided the neural basis of the mechanisms of learning and memory.  In case you may not recall what is the Pavlovian experiment about, here is what he has done in short: He trained dogs by presenting them bell rings and foods together, and the digest system of post-trained dogs are automatically activated as a conditioning reflex even only presented bell rings. As defined in Wikipedia:

Classical conditioning occurs when a conditioned stimulus (CS) is paired with an unconditioned stimulus (US). Usually, the conditioned stimulus is a neutral stimulus (e.g., the sound of a tuning fork), the unconditioned stimulus is biologically potent (e.g., the taste of food) and the unconditioned response (UR) to the unconditioned stimulus is an unlearned reflex response (e.g., salivation). After pairing is repeated (some learning may occur after only one pairing), the organism exhibits a conditioned response (CR) to the conditioned stimulus when the conditioned stimulus is presented alone.

This whole procedure can be analogized as a train-learning process in the neural system, as training is to pairing the CS with the US, and expecting the CR to be as similar as the UR when given the CS alone after the training is done.

Now let’s go back to the Machine Learning part. Artificial Neural Network is a popular topic these days, and you must have heard at least one of its names: Neural Networks, Deep Neural Networks, Deep learning, etc. Among all of its names, I’d prefer one name: Multi-layer Perceptron (MLP), since this name describes the nature of it without exaggeration. It is a mathematical model that resembles a real neural system, which consists connected layers of Perceptrons (as neurons to a neural system) and learns the mapping between inputs (stimulus) and expected outputs (expected responses).

A neuron, aka a nerve cell, that receives, processes, and omits information through electrical or chemical signals. The received signal could be with any form and with any level, however, the neuron only omits either ‘activated’ or ‘not activated’ to others. Just like its biological counterparts, perceptrons take various input signals and only output binary outcomes. However, the difference does exist, since perceptrons deal with digital signals instead of electrical or chemical signals.

Then we can have the definition of perceptron: an algorithm for a binary classifier, which takes input \(x\) and outputs 1 or 0. Think of a scenario, where you are making a decision of whether going out or not on a winter day, depends on the temperature outside.  You can have a function here that takes temperature as \(x\):

\[f(x) = w\cdot x + b\]

where \(w\) represents a weight that decides how important of the temperature change to the decision, and \(b\) as a bias that decides the threshold of ‘going out’ or not. But this is not enough, since it does not explicitly gives you the result of the decision, let’s say, ‘going out’ for 1 and ‘stay home’ for 0. To solve this, we may have an activation function here:

\[g(x) = \left\{\begin{matrix} 1 \textrm{ if }w\cdot x + b> 0 \\ 0 \textrm{ otherwise} \end{matrix}\right.\]

which transforms a continuous signal into a binary one. When combing these two functions, voila, we now have a perceptron!  It now can make the decision of whether going out for you. To make it easy, let’s set \(w\) to 1  and \(b\) to 0 (in real condition, they are learned by the perceptron in a training stage, just like Pavlov trains his dogs 😛 ). Then, the perceptron will tell you that, if the temperature is above 0, go out and have fun; and if not, you better stay due to the severe weather.

Though a perceptron is simple, it would show its tremendous power when linked together like a neural system. This would be a long story, and I am planning to share the story in a ‘friendly’ way. Stay tuned if you are interested, and your opinion needed!



Read More:

3 Replies to “Pavlov’s Dog and Artificial Neural Network (Part 1: Perceptron)”

Leave a Reply

Your email address will not be published. Required fields are marked *