Download CuPit-2 | A Parallel Language for Neural Algorithms: Language
Transcript
12 6 TUTORIAL 2. The network processes these input values and produces output values. 3. These output values are compared to the original output values (i.e., those given in the example) and the dierences are computed. 4. From the dierences, an error measure is computed for each output value. 5. These error measures are fed backwards through the network. 6. During this backward pass through the network, the network uses the error measures to compute changes that should be made to each weight. 7. These changes are accumulated (averaged) over all the examples. This algorithm works for a class of neural networks called \multi layer perceptrons" that will be described in the next section. 6.1.2 The network The central object for any neural network learning algorithm is the network itself. Thus, it is also the central data structure in any program that implements this learning algorithm. The kind of network that is usually used with the backpropagation algorithm is called a \fully connected multi layer perceptron". In the following, we assume a 3 layer perceptron. It consists of three groups of units and two groups of weights. Both kinds of groups (unit groups as well as weight groups) are sometimes called \layers" which makes the term a bit confusing (and has the eect that some people would call our network a 2 layer perceptron). The groups of units are the input units, the hidden units, and the output units. The groups of weights are the hidden weights and the output weights. A weight is a weighted connection between two units. The set of hidden weights contains exactly one connection from each input unit to all of the hidden units. The set of output weights contains exactly one connection from each hidden unit to all of the output units. There is one additional unit, called the bias unit, that constantly outputs the value 1 and that has exactly one connection to each hidden and output unit. 6.1.3 The algorithm The input units do nothing special, they just broadcast the input value they receive (from the example) to all the connections they have to the hidden units. Connections deliver a value they receive at one end to the other end after multiplying it with the weight that is stored in the connection. Hidden and output units accumulate the values delivered by the connections that they receive and apply a so-called activation function to the result in order to compute their output value. In the case of hidden units this output value is again broadcasted to the outgoing connections. In the case of output units the output value is not broadcasted but instead compared with the correct output value given in the current example. The dierence of these two values is squared and the result used as the error value for this example at any single unit. The process so far is called the forward pass through the network. The error values are then propagated backwards through the network. This process is called the backward pass and gives the whole algorithm its name. The purpose of the backward propagation of errors is that each weight can compute its contribution to the overall error for this example. From this contribution, each weight can compute a delta which describes how the weight should change in order to reduce the error most. The output weights multiply the errors by their weight values and propagate the product to the hidden units. The hidden units accumulate these weighted errors and multiply the sum with the derivative of the activation function of the value they received as input in the forward pass. This hidden unit error value is further propagated backwards to the hidden weights, which operate on them just like the output weights did on the output errors. This combination of forward pass and backward pass is repeated for each example. The connections accumulate the delta values they compute in each backward pass. When all examples are processed, each connection weight adapts by adding the product of its accumulated deltas and a parameter called the learning rate to its current weight value. This example processing and weight adaption is repeated until some custom termination criterion is fullled.