Connection between BPTT and RTRL
*No subscription required
This paper shows the connection between the Backpropagation Through Time (BPTT) algorithm, its truncated forms with truncation depth h, and the Recurrent Real Time Learning (RTRL) algorithm. The comparison is done by looking at a fully connected recurrent network, which is based on the same error function and calculations, using exact ordered derivatives. Two kind of formulas, based on total ordered derivatives, for BPTT(h) are given and proven to be equivalent. Of the two formulae, the second one, can be interpreted by a target modification in the case of h. Further, a combination of BPTT and RTRL is proposed to account for possible instabilities caused by weight adaptation. An overview of BPTT and RTRL and their implementations as well as their interpretations and uses are outlined.
|Publication Type:||Journal Article|
|Publisher:||World Scientific and Engineering Academy and Society|
|Item Control Page|