Lecture 7B: Introduction to Neural Networks: RBF, MLP, & Backpropagation (2020-03-30)

In this lecture, we continue our introduction to artificial neural networks (ANN). We start with a review of a model of a single neuron can be viewed as a generalized linear model (GLM), and how a single-layer (single-neuron) perceptron (SLP) can be a binary classifier for linearly separable data sets. We then discuss how radial basis function (RBF) neural networks (RBFNN) can be viewed as nonlinear transformations of data that allow for linear separability in the new space. These RBF's combine multiple neurons together, which gives us an opportunity to introduce the multi-layer perceptron (MLP) and closely related variants (like the convolutional neural network, CNN). We discuss the Universal Approximation Theorem (UAT) and what it means for why we bother with deep neural networks. We then start to introduce backpropagation as a way of training MLP's. 

Whiteboard notes for this lecture can be found at:

https://www.dropbox.com/s/qe3465z0tb3vds0/IEE598-Lecture7B-Notes.pdf?dl=0

Lecture 7A: Introduction to Neural Networks (2020-03-25)

This lectures introduces the very basics of neural networks, from a description of a basic neuron from biology to the simple single-layer perceptron that it inspired. We cover the relationships between a single-layer perceptron and generalized linear modeling (GLM). We then demonstrate how the weights of a single artificial neuron can be trained for binary classification problems (with linearly separable data).

Whiteboard notes for this lecture can be found at:

https://www.dropbox.com/s/bswuk0gnqyk6hwe/IEE598-Lecture7A-Notes.pdf?dl=0

Lecture 6D: Distributed AI and Swarm Intelligence, Part 4 - Particle Swarm Optimization (PSO) and friends (2020-03-23)

In this lecture, we continue to cover basic Particle Swarm Optimization (PSO) and some slight variations on the basic model that help improve performance and practicality of the approach. We use the social aggregation in PSO to motivate topics that will come up in the next few lectures on opinion dynamics and related dynamical phenomena that are less related to optimization and more related to consensus and stability in distributed groups.

The whiteboard notes for this lecture can be found at: https://www.dropbox.com/s/dmzjecmuxzzv3xk/IEE598-Lecture6D-Notes.pdf?dl=0

Lecture 6C: Distributed AI and Swarm Intelligence, Part 3 - Classical Particle Swarm Optimization (PSO) & Its Motivations (2020-03-08)

This lecture continues our introduction to Swarm Intelligence algorithms, with a major focus on Particle Swarm Optimization (PSO) and the models of collective motion that influenced it (like Reynolds' Boids model for computer graphics, Vicsek self-propelled particles model, and "selfish herds" in general). We contrast PSO with more "stigmergic" optimization algorithms like ant colony optimization (ACO) and bacterial foraging optimization (BFO) and view this more as an algorithm that tries to maintain cohesion of a group of individuals searching for good positions within a space.

Whiteboard notes for this lecture can be found at:

https://www.dropbox.com/s/cymeqthkbs1mbb1/IEE598-Lecture6C-Notes.pdf?dl=0

Lecture 6B: Distributed AI and Swarm Intelligence, Part 2 - Bacterial Foraging Optimization and Intro to Particle Swarm Optimization (PSO) (2020-03-16)

In this lecture, we continue our discussion of algorithms in the category of swarm intelligence. We start with a recap of Ant System (AS) – a precursor to Ant Colony Optimization (ACO) with very similar underlying mechanisms of stigmergic operation based on virtual pheromones. We then pivot to describe bacterial foraging optimization (BFO), which considers freely moving bacteria that can "tumble" whenever reaching a nutrient decline and "run" across nutrient increases. Like trail-laying ants, these bacteria can use chemicals to aggregate and spread social information. These chemicals can be attractive or repulsive. Like evolutionary algorithms, these bacteria can be eliminated based on poor local performance and increased in areas based on good local performance. After covering bacterial foraging optimization, we start to introduce particle swarm optimization (PSO), which is a similar motion-based metaheuristic meant to maintain cohesive swarms that move through a decision space. We will pick up on PSO more in the next lecture.

Whiteboard lecture notes (with some corrections) can be found at:

https://www.dropbox.com/s/2mkcuyl8kzvfev1/IEE598-Lecture6B-Notes.pdf?dl=0

Lecture 6A: Distributed AI and Swarm Intelligence, Part 1 - Ant Colony Optimization (ACO) (2020-03-04)

This lecture introduces distributed AI and swarm intelligence. After a brief discussion of the difference between these approaches and the previous evolutionary and nature-inspired metaheuristics, we cover Ant Colony Optimization (ACO)/Ant System (AS), which is an approach for combinatorial optimization over discrete decision spaces based on how trail-laying ants find the shortest path from nest to food. Because this approach involves agents communicating/coordinating through persistent modifications of the environment, it falls under the category of using "stigmergy."

Whiteboard notes can be found at:

https://www.dropbox.com/s/8vmyqmpqd7d5srr/IEE598-Lecture6A-2020-03-04-15-08-05.mp3?dl=0

Lecture 5C: From MCMC Sampling to Optimization by Simulated Annealing (2020-03-02)

In this lecture, we start with an outline of the Metropolis–Hastings (MH) Markov Chain Monte Carlo (MCMC) sampling algorithm and describe how instantiating it with the Boltzmann–Gibbs distribution (which turns it into the classical Metropolis algorithm) is equivalent to sampling from an abstract energy landscape in the "least biased" way given a kinetic energy budget. We then discuss how the acceptance likelihood ratio allows for this (and other) MCMC algorithms to be used in applications where the desired relative frequency of outcomes is known but the normalization constant usually needed for a probability density is cumbersome to calculate. From there, we move from MH to Simulated Annealing (SA), which is a special case of the Metropolis algorithm with an annealing schedule. SA sets a temperature and then samples in the "least biased" (maximum entropy) way for that temperature before decreasing temperature, in the hopes of eventually finding the minimum of an optimization objective.

Whiteboard notes from this lecture can be found at: https://www.dropbox.com/s/hprzvgrshg755ge/IEE598-Lecture5C-Notes.pdf?dl=0