Generative Methods in AI for Science
Module aims
This module addresses the fundamental concepts and advanced methodologies of generative models and relates them to real-world problems in a variety of domains in AI 4 Science. The aim is to provide an overview of different approaches, both classical and emerging. The module will equip you with the necessary knowledge and skills to work to participate robustly in the breadth of topics ranging from protein generative models, to sampling from Boltzmann distribution, and beyond.
Learning outcomes
On successful completion of this module, students will be able to:
1. Demonstrate a comprehensive understanding of advanced generative modelling methods, including autoregressive models, normalizing flows, Boltzmann generators, diffusion models, score-based models, flow matching models, and equivariant generative models.
2. Formulate scientific generation problems as probabilistic modelling, sampling, or reward-tilted distribution learning problems.
3. Understand the role of physical and geometric constraints in scientific generative modelling, including permutation symmetry, translation symmetry, rotation symmetry, E(3)-equivariance, SE(3)-equivariance, periodicity, lattice symmetries, and space-group structure.
4. Explain the relationship between generative modelling and statistical mechanics, including Boltzmann distributions, potential energy functions, free energy, molecular dynamics, Langevin dynamics, Markov chain Monte Carlo, importance sampling, and effective sample size.
5. Apply inference-time steering methods, including guidance, reward tilting, best-of-N sampling, sequential Monte Carlo, Feynman–Kac steering, and search-based methods, to improve scientific generative models.
Module syllabus
1. Generative modelling and scientific data
This part of the module introduces generative modelling as the problem of learning and sampling from complex data distributions. Students will begin with familiar autoregressive models and then examine why scientific domains require richer representations and constraints.
2. Autoregressive models, Normalizing Flows for scientific sequences
This part covers autoregressive models as a starting point for scientific generative modelling. Students will study the strengths and limitations of sequence-based approaches for molecules, proteins, and other scientific objects.
Additionally, we will cover normalizing flows as exact-likelihood generative models. Students will learn how invertible transformations can map simple base distributions to complex scientific distributions while retaining tractable density evaluation.
3. Boltzmann distributions and Boltzmann generators
This part connects generative modelling to statistical mechanics. Students will study Boltzmann distributions as target distributions for molecular systems and learn how normalizing flows can be trained as amortized samplers.
4. Dynamic measure transport: diffusion, score models, and flow matching
This part introduces modern generative models as methods for transporting probability measures through time. Students will study diffusion models, score-based models, continuous normalizing flows, flow matching, rectified flows, and stochastic interpolants under a unified view of dynamic measure transport.
5. Dynamic measure transport for molecules, proteins, and materials
This part studies how diffusion, score-based models, and flow matching are adapted to scientific objects. Students will learn how representation, geometry, symmetry, and physical constraints affect the design of noising processes, probability paths, score networks, velocity fields, and samplers.
6. Flow maps, consistency, and fast scientific generation
This part studies methods for reducing the sampling cost of dynamic generative models. Students will learn about flow maps, consistency models, progressive distillation, mean flows, shortcut models, and few-step generation.
7. Symmetry, invariance, and equivariance
This part introduces the symmetry principles required for scientific generative modelling. Students will learn why physical and geometric symmetries should be built into models of molecules, proteins, and materials.
8. Molecular dynamics, MCMC, and statistical mechanics
This part grounds the course in classical scientific simulation. Students will learn how molecular dynamics, MCMC, and statistical mechanics define the distributions that many scientific generative models aim to approximate or accelerate.
9. Inference-time scaling I: steering and guided generation
This part studies how pretrained generative models can be steered at inference time toward desired scientific properties. Students will learn how conditional generation, reward guidance, energy guidance, reranking, and particle-based methods can improve scientific design without necessarily changing model parameters.
Teaching methods
This is an advanced module and requires adequate mathematical maturity to participate. The material covered is substantive but any motivated student at the end of the course will be able to conduct research in this field immediately. Consequently, the focus of the course is aimed to train junior researchers and as a result a large component of the course requires independent reading of the papers in the weekly reading list. The course content will provide a sufficient introduction to the topics but the reading list will require going beyond the presented lectures. The material in the course is supplanted by bi-weekly assignments that are intended to test the concepts learned from both the in class material and papers in a way a research sprint of 2 weeks can be done. Students will receive support through GTAs every week during the last 1hr of the second lecture.
It is essential that students are familiar with Deep Learning and have taken Mathematics for Machine Learning prior to this course.
An online service will be used as an open discussion forum for the module.
Assessments
This module will be broken down into 4 assignments that occur at a bi-weekly cadence. These assignments are designed to be challenging and open ended and students are encouraged to form groups of 1-3 people. The assignments will account for 20% of the grades. The remaining 80% will be assessed via a traditional exam.
There will be written feedback for the assessed coursework and class-wide feedback explaining common pitfalls and suggestions for improvement.
The GTAs and instructor will be available during every lecture to answer questions related to research.