In this lecture, we finish up our coverage of supervised learning of feedforward multi-layer perceptrons with a discussion of how the Convolutional Neural Network imposes an inductive bias that simplifies training and pays off for images but may not work so well for text strings. We then shift our focus to recurrent networks with temporal supervision, which may help to provide a solution when highly local inductive biases aren't effective (as in for text and time-series analysis). We discuss several coincidence detectors from neuroscience in the context of hearing and vision, and we use them to motivate Time Delay Neural Networks (TDNNs) as our bridge to Recurrent Neural Networks (RNNs). This allows for analogies to be made to Finite Impulse Response (FIR) and Infinite Impulse Response (IIR) filters. We close by transitioning from a basic output-feedback configuration to a generic RNN with hidden states but effectively no "layers." We will pick up next time with backpropagation-through-time (BPTT), Long Short Term Memory (LSTM), reservoir computing (Echo State Networks, ESN's), and an introduction to reinforcement learning. Interactive demonstration widgets related to this lecture can be found at:
- Toward Multimodal AI: https://tpavlic.github.io/asu-bioinspired-ai-and-optimization/transformers/toward_multimodal_AI.html
- RNN Explorer: https://tpavlic.github.io/asu-bioinspired-ai-and-optimization/recurrent_neural_networks/rnn_explorer.html
No comments:
Post a Comment