Lecture 8A (2025-04-29): Complex Systems Models of Computation – Cellular Automata and Neighbors

This lecture introduces approaches for understanding (and building) computational systems that emerge out of Complex Adaptive Systems (CAS). It first motivates the idea that systems of many, interconnected parts that each are relatively easy to understand in isolation can come together in a system whose network of interactions leads to emergent global phenomena that cannot be predicted from the properties or behaviors of any individual component. We then focus on the role of space in the functions and properties that emerge at a global level. We do this through the example of the Interacting Particle System (IPS) known as the "voter model", which can be viewed as a model for neutral evolution in spatially structured populations. We show that the dual process for the Voter Model is a time-reversed set of coalescing random walkers for which consensus in the model corresponds to whether walkers are sure to coalesce into a single walker in the past of the dual process. This lets us apply Pólya's recurrence theorem and show that consensus is guaranteed (with probability 1) for 1- and 2-dimensional lattices but not guaranteed for lattices of 3 dimensions or higher. This implies that neutral evolution (for example) in a 3D spatial structure may not always lead to fixation on one genotype. We then pivot to introducing Elementary Cellular Automata (ECA) and describe a few rules that demonstrate how they work. We close the regular lecture by connecting CA's back to neural networks (the previous unit) and evolutionary algorithms (the first unit), thus introducing the Cellular Evolutionary Algorithm (cEA). We then extend the lecture a little longer than usual in order to do a demonstration of several ECA's in NetLogo, including a demonstration of how to combine two ECA rules to generate a reliable density classifier.

Whiteboard notes for this lecture can be found at:
https://www.dropbox.com/scl/fi/gfna3a2aj49fq2a4sovst/IEE598-Lecture8A-2025-04-29-Complex_Systems_Models_of_Computation-Cellular_Automata_and_Neighbors-Notes.pdf?rlkey=mo5jag4axljxrbpnq6wkk9egh&dl=0

Lecture 7F (2025-04-24): Spiking Neural Networks and Neuromorphic Computation

This lecture explores how real and artificial brains learn using spikes. We begin by reviewing the structure and behavior of spiking neurons, focusing on the Leaky Integrate-and-Fire (LIF) model and the efficiency of sparse, event-driven temporal coding. We then introduce Spike-Timing-Dependent Plasticity (STDP), a biologically inspired learning rule that adjusts synaptic strength based on the relative timing of spikes. From there, we survey major neuromorphic hardware platforms—SpiNNaker, TrueNorth, and Loihi—highlighting their architectural differences and support for learning. We then examine memristor-based crossbar arrays as an analog substrate for STDP, including a case study from Boyn et al. (2017). Finally, we return to Hebbian learning as a conceptual foundation ("fire together, wire together") and explore how simple local decentralized unsupervised Hebbian-like learning rules for conventional ANNs can also produce meaningful clustering behavior. We close with a discussion of future directions, including neuromodulation, synaptic adaptability, and recent research on using sleep-inspired replay to prevent catastrophic forgetting in spiking neural networks.

Whiteboard notes for this lecture can be found at:
https://www.dropbox.com/scl/fi/8mqjreoitin3qadzk9ofm/IEE598-Lecture7F-2025-04-24-Spiking_Neural_Networks_and_Neuromorphic_Computation-Notes.pdf?rlkey=l83a286aig0fpibafuvofr0hc&dl=0

Lecture 7E (2025-04-22): Learning without a Teacher – Unsupervised and Self-Supervised Learning

This lecture covers unsupervised and self-supervised learning, focusing on how both brains and machines discover structure without external labels or rewards (akin to non-associative learning). It begins with examples of unsupervised learning, including clustering, principal component analysis, and autoencoders, and then explores how biological systems like the olfactory pathway in insects organize complex sensory input into compressed, low-dimensional codes. We take a detailed look at the structure of the honeybee brain, examining how floral odors are transformed through the antennal lobe’s glomerular code into organized neural representations. We then transition into self-supervised learning (akin to latent learning) by introducing predictive coding and sensorimotor prediction, highlighting how brains use internal models to anticipate and correct sensory input. Finally, we close by discussing how modern AI systems like GPT (and BERT) leverage self-supervised objectives to build rich internal representations from raw data.

Whiteboard notes for this lecture can be found at:
https://www.dropbox.com/scl/fi/qwezfleqplmxtiobfpoew/IEE598-Lecture7E-2025-04-22-Learning_without_a_Teacher-Unsupervised_and_Self-Supervised_Learning-Notes.pdf?rlkey=4k5o8j8no3s9x7xc5di676qz3&dl=0

Lecture 7D (2025-04-17): Reinforcement Learning – Active Learning in Rewarding Environments

In this lecture, we introduce reinforcement learning (RL) with motivations from animal behavior and connections to optimization metaheurisitcs such as Ant Colony Optimization (ACO) and Simulated Annealing (SA). We start by returning to a simple model of pheromone-trail-based foraging by ants (reminiscent of Ant Colony Optimization (ACO)) and formalize the components of the ant action in terms of quality tables for (state, action) pairs, as would be used in RL. We then introduce the quality Q(s,a) function and Q-learning, including two different methods of exploration (epsilon-greedy and softmax) with connections to how different species of ants respond to pheromones. We discuss Deep Q Networks (DQN's), as a connection to neural networks, and then move on to motivating an interpretation of the discount factor using Charnov's Marginal Value Theorem (MVT) of optimal foraging theory (OFT). We close with a discussion of the Matching Law from psychology and how a group of RL agents will converge to a social version of the Matching Law, the Ideal Free Distribution (IFD). Next time, we will cover unsupervised and self-supervised learning, which are approaches where the learning happens even without reward.

Whiteboard notes for this lecture can be found at:
https://www.dropbox.com/scl/fi/gyux79ukkcs0n7buizfr1/IEE598-Lecture7D-2025-04-17-Reinforcement_Learning-Active_Learning_in_Rewarding_Environments-Notes.pdf?rlkey=ix5qf4a5yz97ppsx97h6sphao&dl=0

Lecture 7C (2025-04-15): Recurrent Networks and Temporal Supervision

This lecture focuses on Recurrent Neural Networks (RNNs), which leverage delays within neural networks as storage elements that can be used to make inferences about temporal patterns. We start with an overview of coincidence detectors thought to be used for spatial localization and motion detection in the auditory (Jeffress model) and visual systems (Hassenstein–Reichardt model). This motivates the introduction of Time Delay Neural Networks (TDNNs), which generalize the use of delay lines used in the coincidence-detection circuits. We show how feed-forward TDNNs can be used to identify finite-duration patterns (where the number of neural elements necessary must scale up with the length of the pattern) and draw connections to Finite Impulse Response (FIR) filters. Then we shift to Recurrent Neural Networks and draw analogies to Infinite Impulse Response (IIR) filters that are able to identify patterns over very long durations of input while only using a few neurons (leveraging the implicit memory in the output state(s)). That brings us to Long Short-Term Memory (LSTM) (and the Gated Recurrent Unit, GRU), which is a popular form of RNN that has become less emphasized since the growth in the use of Transformers. We close by showing that randomly weighted, fixed RNN's can be used as "reservoirs" in Echo State Networks as feature extractors that spread out temporal patterns over space, allowing for simple feed-forward decoders (and possibly multiple of them sharing the same reservoir decoder resource) to do complex time-series analysis. These reservoirs can also be instantiated in other dynamical media, such as actual water reservoirs and even materials embedded within soft robots – each of these examples fit within the larger area of "Reservoir Machines" or "Reservoir Computing." Next time, we focus on Reinforcement Learning and its connections to animal foraging.

Whiteboard notes for this lecture can be found at:
https://www.dropbox.com/scl/fi/u5qjwvqwwb6ok378kw2lm/IEE598-Lecture7C-2025-04-15-Recurrent_Networks_and_Temporal_Supervision-Notes.pdf?rlkey=gqhyq06wdzpw0m6t4fo1hfxqe&dl=0

Lecture 7B (2025-04-10): Feeding Forward from Neurons to Networks

In this lecture, we present the foundations of supervised learning of feedforward neural networks, starting with the original inspiration from basic models of the neuron. We start with a review of the activation of a basic neuron and then map that to the Single Layer Perceptron (SLP), which we describe as a tool for binary classification of linearly separable data. Then, to extend these capabilities to data sets that are not linearly separable, we introduce Radial Basis Function Neural Networks (RBFNNs), whose Receptive Field Units (RFUs) act as a hidden layer that allow the RBFNN to do much more than the SLP. Thus, the RBFNN is our first example of a single-hidden-layer neural network. This gives us an opportunity to discuss Universal Approximation Theorems (UAT's) which help to explain why the RBFNN is so much more capable than the SLP. Despite its strengths, the RBFNN is not convenient to train. So, guided by UAT, we introduce the Multi-Layer Perceptron (MLP), which is a generalized version of the SLP that includes a hidden layer whose non-linear activation functions allow for universal approximation. We discuss how backpropagation can be used to train MLP's efficiently so long as activation functions are differentiable. We close with an introduction to Convolutional Neural Networks (CNNs) as an MLP that implements receptive fields very similar to RBFNN but in a way that is more flexible/trainable than the RBFNN. Next time, we will start discussing recurrent neural networks, their connection to biology, and how to train them.

Whiteboard notes for this lecture are available at:
https://www.dropbox.com/scl/fi/t23j4gupde7zyue83guz3/IEE598-Lecture7B-2025-04-10-Feeding_Forward_from_Neurons_to_Networks-Notes.pdf?rlkey=vpyd1htoswq54reb892clgch4&dl=0

Lecture 7A (2025-04-08): Neural Foundations of Learning

This lecture opens the unit on Neural Computation and Learning, which discusses the neurobiological underpinnings of learning in biological systems and attempts at developing similar capabilities in artificial systems with artificial neural networks (including spiking neural networks). In this lecture, learning, memory, and neuroplasticity are introduced alongside a basic model of the canonical neuron with axons, dendrites, and action potentials. Time is spent discussing the differences in the function and costs of working, short-term, and long-term memory as well as the three different neuronal mechanisms underlying these three different forms of memory. Then the basics of biological learning, from sensory adaptation to non-associative learning (habituation and sensitization) to associative learning/conditioning (classical and operant) to latent learning, are presented along with basic models of how these different forms of learning can be built up from mechanisms of neuroplasticity discussed earlier. Finally, the different modalities of machine learning (unsupervised, self-supervised, reinforcement, and supervised learning) are presented and connected to the best-fitting biological learning paradigms (as well as potential neuronal mechanisms that could be used ot build such capabilities).

Whiteboard notes for this lecture can be found at:
https://www.dropbox.com/scl/fi/mlw3u4yf19fbqnypjc39z/IEE598-Lecture7A-2025-04-08-Neural_Foundations_of_Learning-Notes.pdf?rlkey=5mhnsfmmgtiad050u69qov80x&dl=0

Lecture 6C (2025-04-03): Distributed AI and Swarm Intelligence, Part 3 – Bacterial Foraging Optimization (BFO) and Particle Swarm Optimization (PSO)

In this lecture, we discuss two other examples of Swarm Intelligence applied to the numerical optimization space – Bacterial Foraging Optimization (BFO) and Particle Swarm Optimization (PSO). Both of these make use of a population moving through an unconstrained environment that respond to both their local experience of an optimization objective as well as the positions of others around them. In BFO, which is inspired by the "run-and-tumble" behavior of certain kinds of flagellated bacteria, bacteria attempt to minimize exposure to a surrounding chemical (the optimization objective to minimize) while also responding to attractive and repellant chemicals emitted by all other bacteria (that have an intensity that rolls off with distance from the emitter). Bacteria "run" in a consistent direction either for a specified amount of algorithm steps or when the sensed cost (from both the optimization objective and the attractant and repellant signals) increases, at which point they "tumble" to a new random direction. After the "lifetime" of all bacteria passes, half of them with the most cumulative exposure die and are replaced by clones of the other half. At certain points, bacteria get randomly moved to other parts of the decision space regardless of how effective their current search has been. This process makes BFO computationally costly but very effective at global optimization and even tracking problems (so long as the reference objective to track is changing slowly enough). PSO has a much simpler implementation where each self-propelled particle has a position and a velocity and an "inertia" that prevents rapid changes in heading. Here, each SPP in PSO remembers the position of its best location as well as the best position of every other SPP in the swarm, and these two positions act like attractors that pull the particle to them. Both PSO and BFO are swarm-intelligence algorithms, but neither incorporate the notion of "stigmergy" (the indirect coordination of individuals through modifications and responsiveness to the shared environment). ACO, which was discussed earlier, is an example of stigmergy (as it maintains a matrix of long-lasting chemical depositions that have their own state, independent of the "ants" that deposited them and the ants that respond to them).

Whiteboard notes for this lecture can be found at:
https://www.dropbox.com/scl/fi/2opklen1vwap6qx1v3vby/IEE598-Lecture6C-2025-04-03-Bacterial_Foraging_Optimization_BFO_and_Particle_Swarm_Optimization_PSO-Notes.pdf?rlkey=tn8ghxsrhxvuip1xq5drd8vte&dl=0

Lecture 6B (2025-04-01): Distributed AI and Swarm Intelligence, Part 2 – ACO and Introduction to Bacterial Foraging Optimization (BFO)

This lecture primarily focuses on describing the structure and operation of Ant System (AS), a precursor to the Ant Colon Optimization (ACO) metaheuristic. It discusses the key algorithmic components of the approach and briefly describes the ant trail-laying behavior that inspired it. At the end of the lecture, Bacterial Foraging Optimization (BFO) is introduced. A full treatment of BFO will be covered in the next lecture.

Whiteboard notes for this lecture can be found at:
https://www.dropbox.com/scl/fi/oreo91lww327dcbtwq6e3/IEE598-Lecture6B-2025-04-01-ACO_and_Introduction_to_Bacterial_Foraging_Optimization_BFO-Notes.pdf?rlkey=66jx4pfv369kbcngicntv3lem&dl=0