IEE/CSE 598: Bio-Inspired AI and Optimization: Lecture 7D (2026-04-21): RNN's – Backpropagation Through Time (BPTT), Long Short Term Memory (LSTM), and Reservoir Computing/Echo State Networks (ESNs)

Tuesday, April 21, 2026

Lecture 7D (2026-04-21): RNN's – Backpropagation Through Time (BPTT), Long Short Term Memory (LSTM), and Reservoir Computing/Echo State Networks (ESNs)

In this lecture, we continue our discussion of Recurrent Neural Networks (RNN's) as generalized forms of Time Delay Neural Network (TDNN) that can do time-series classification (and prediction) using an inductive bias that can pull in information from a wide range of times (well beyond the simple size of the neural network, due to the use of output feedback to maintain state). We discuss how these networks can be trained with Backpropagation Through Time (BPTT) and some limitations of this approach. This motivates the more constrained Long Short Term Memory (LSTM) and Gated Recurrent Unit (GRU) architectures, which mitigate some issues with training general RNN's. We then pivot to a different approach entirely -- using recurrent neural networks as untrained "reservoirs" whose outputs are dynamical encoders that spread out temporal patterns into spatial ones that can be learned with a single-layer perceptron. We demonstrate this using an Echo State Network (ESN) and walk through how even small networks can provide significant separability for time series. We also have a discussion of how these approaches can be used for predicting chaotic time series, with applications in finance as well as digital twins (e.g., for manufacturing systems).

Interactive demonstrations connected to this lecture can be found at: