Learning Rate<\/td> 0.01<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\nThen the gradient descent algorithm will pick the next point 0.025 away from the previous point. A small learning rate will take too long and a very large learning rate the algorithm might diverge away from the minimum point (miss the minimum completely).<\/p>\n\n\n\n
Finally, the weights are updated incrementally after each epoch (pass over the training dataset) till we get the best results.<\/p>\n\n\n\n
Stochastic Gradient Descent<\/h3>\n\n\n\n In gradient descent, a batch is the total number of examples you use to calculate the gradient in a single iteration. So far, we have assumed that the batch has been the entire data set. But for large datasets, the gradient computation might be expensive.
Back-propagation<\/h3>\n\n\n\n Back-propagation is simply a technique or method of updating the weights. We are aware of partial derivatives, chain rule and most importantly gradient descent. But with Neural networks having multiple layers and different activation functions make it difficult to visualize how everything comes together. Consider, a simple example with the following architecture:<\/p>\n\n\n\n<\/figcaption><\/figure>\n\n\n\nForward Pass<\/h4>\n\n\n\n Step 1:<\/strong> Initialization Let us initialize the weights and the bias.Table 1 a: Weight Initialization Example<\/p>\n\n\n\nWeights<\/th> Value<\/th><\/tr><\/thead> w1<\/td> 0.10<\/td><\/tr> w2<\/td> 0.15<\/td><\/tr> w3<\/td> 0.03<\/td><\/tr> w4<\/td> 0.08<\/td><\/tr> w5<\/td> 0.18<\/td><\/tr> w6<\/td> 0.06<\/td><\/tr> w7<\/td> 0.11<\/td><\/tr> w8<\/td> 0.26<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\nTable 1: Dominated\/Non-Dominated Example<\/p>\n\n\n\nBias<\/th> Value<\/th><\/tr><\/thead> b1<\/td> 0.05<\/td><\/tr> b2<\/td> 0.42<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\nAssume take the initial input values to be [0.95,0.06]<\/strong> and the target value [0.05,0.82]<\/strong>.<\/p>\n\n\n\nStep 2:<\/strong> Calculations<\/p>\n\n\n\nTo get the value of H1:<\/p>\n\n\n\n
H1 = w1 * x1 + w2 * x2 + b1\n = 0.1 * 0.95 + 0.15 * 0.06 + 0.05\n = 0.154\n<\/pre>\n\n\n\nAs we have a sigmoid activation function:<\/p>\n\n\n
$
Similarly, we can calculate H2.<\/p>\n\n\n\n
H1 = 0.538 and H2 = 0.52<\/p><\/blockquote>\n\n\n\n
Now we calculate the value for output nodes Y1 and Y2.<\/p>\n\n\n\n
Y1 = w5 * H1 + w6 * H2 + b2\n = 0.18 * 0.538 + 0.06 * 0.52 + 0.42\n = 0.548\n<\/pre>\n\n\n$
Upon calculation:<\/p>\n\n\n\n
Y1 = 0.633 & Y2 = 0.648<\/p><\/blockquote>\n\n\n\n
Step 3:<\/strong> Error Function Let the error function be:<\/p>\n\n\n$
Total Error (E) = E1 + E2 = 0.184972\nE1 = 0.5 * (0.05 - 0.63368)^2 = 0.17\nE2 = 0.5 * (0.82 - 0.64893)^2 = 0.014 \n<\/pre>\n\n\n\nBackward Pass<\/h4>\n\n\n\n Back-propagate the Errors to update the weights.<\/p>\n\n\n\n
Error at W5:<\/p>\n\n\n
$
$
Component 1: The Cost\/Error Function<\/p>\n\n\n\n
target: T\noutput: out\nE = 0.5 * (T1 - out Y1)^2 + 0.5 * (T2 - out Y2)^2\nDifferentiating:\n- (T1 - out Y1) = - (0.05 - 0.63368) = 0.58368\n<\/pre>\n\n\n\nComponent 2: The Activation function<\/p>\n\n\n\n
output: out\nout Y1 = 1\/(1 + exp(-Y1))\nDifferentiating:\nout Y1 * (1 - out Y1) = 0.63368 * (1 - 0.63368) = 0.23213\n<\/pre>\n\n\n\nComponent 3: The Function of Weights<\/p>\n\n\n\n
Y1 = w5 * H1 + w6 * H2 + b2\nDifferentiating:\nH1 * 1 = 0.538\n<\/pre>\n\n\n\nFinally, we have the change in W5:<\/p>\n\n\n
$ \\partial E \\over \\partial W5
=0.58368\u22170.23213\u22170.538<\/code><\/p>\n\n\n\n=0.07289<\/code><\/p>\n\n\n\nIn order to update W5 recall the discussion on gradient descent. Let alpha be learning rate with a chosen value of 0.01.<\/p>\n\n\n\n
Updated W5 will be:<\/p><\/blockquote>\n\n\n
$
=0.18+0.01\u22170.07289<\/code><\/p>\n\n\n\n=0.1807289<\/code><\/p>\n\n\n\nSimilarly, we can update the remaining weights. Let\u2019s have a look at the formula to update W1:<\/p>\n\n\n
\\frac{\\partial E}{\\partial w1}<\/p>\n\n\n\n
equals<\/p><\/blockquote>\n\n\n
$<\/figcaption><\/figure>\n\n\n\nIt feels like it is complicated, but really we are going back layer by layer to get the respective value. As w1 feeds into neuron H1 and H1 is connected to Y1 and Y2. Moving backwards, we are differentiating the error function following which Y1 and Y2 (the activation function and the function of Weights) . That leads us to H1 where we differentiate its activation function and its respective function of weights.<\/p>\n\n\n\n
This is how we back-propagate the errors and update all the weights. Once we update all the weights, that is one epoch<\/strong> or pass over the dataset. Further, we start the entire process of forward pass and backward pass again. This process is repeated for multiple times with the purpose of minimizing error.<\/p>\n\n\n\nWhen do we stop?<\/p><\/blockquote>\n\n\n\n
We stop prior to over-fitting that is we want the minimum validation error but we do not want the training error to be lower than the validation error.<\/p>\n\n\n\n
Hopefully, this explains the entire process of how neural networks actually work and sheds some light on gradient descent and back-propagation.<\/p>\n\n\n\n
What\u2019s Next?<\/h3>\n\n\n\n Activation<\/a>: We have talked about activation functions in the past posts, but let\u2019s understand in more detail the different types of activation functions and explore their characteristics.<\/p>\n","protected":false},"excerpt":{"rendered":"Understand the concept of Gradient Descent and Back-propagation to get some idea of how Neural Networks work. Deep Learning Series In this series, with each blog post we dive deeper into the world of deep learning and along the way build intuition and understanding. This series has been inspired from and references several sources. It […]<\/p>\n","protected":false},"author":1,"featured_media":549,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_mi_skip_tracking":false,"spay_email":"","jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true},"categories":[72,111],"tags":[107,104,80,106,108,75,109,105],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/datasciencediscovery.com\/wp-content\/uploads\/2019\/09\/brain-gears.jpg?fit=1280%2C853&ssl=1","jetpack_publicize_connections":[],"_links":{"self":[{"href":"https:\/\/datasciencediscovery.com\/index.php\/wp-json\/wp\/v2\/posts\/511"}],"collection":[{"href":"https:\/\/datasciencediscovery.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/datasciencediscovery.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/datasciencediscovery.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/datasciencediscovery.com\/index.php\/wp-json\/wp\/v2\/comments?post=511"}],"version-history":[{"count":12,"href":"https:\/\/datasciencediscovery.com\/index.php\/wp-json\/wp\/v2\/posts\/511\/revisions"}],"predecessor-version":[{"id":803,"href":"https:\/\/datasciencediscovery.com\/index.php\/wp-json\/wp\/v2\/posts\/511\/revisions\/803"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/datasciencediscovery.com\/index.php\/wp-json\/wp\/v2\/media\/549"}],"wp:attachment":[{"href":"https:\/\/datasciencediscovery.com\/index.php\/wp-json\/wp\/v2\/media?parent=511"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/datasciencediscovery.com\/index.php\/wp-json\/wp\/v2\/categories?post=511"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/datasciencediscovery.com\/index.php\/wp-json\/wp\/v2\/tags?post=511"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
![\"gradient](\"https:\/\/i0.wp.com\/datasciencediscovery.com\/wp-content\/uploads\/2018\/12\/Gradient_Descent.jpg?resize=291%2C300\" "\\"\\"")
![\"chain](\"https:\/\/i0.wp.com\/datasciencediscovery.com\/wp-content\/uploads\/2018\/12\/chain_rule.jpg?resize=118%2C45\" "\\"\\"")
![\"back](\"https:\/\/i0.wp.com\/datasciencediscovery.com\/wp-content\/uploads\/2018\/12\/backprop_chain_rule.jpg?resize=370%2C74\" "\\"\\"")
![\"neural](\"https:\/\/i0.wp.com\/datasciencediscovery.com\/wp-content\/uploads\/2018\/12\/neural_network_layer.jpg?resize=791%2C388\" "\\"\\"")