Causality and Explanation in AI
Module aims
As AI systems are deployed in increasingly consequential settings, the ability to understand and explain why models behave as they do has become essential for responsible practice. This module equips students with tools from explainable AI and causality to address these challenges. The module divides into three parts. The first, on explainable AI (XAI), concerns approaches for understanding why a given AI has reached specific decisions, explanations being essential for assessing whether a system's behaviour is trustworthy and justified. This motivates the second part, on causal reasoning, which introduces Pearl's causal hierarchy and develops the formal machinery for reasoning about interventions and counterfactuals. The third, on learning and decision-making with causality, concerns the inference of causal structure from data and causal approaches to decision-making.
Learning outcomes
Upon completion of the module, students will be able to:
1. Critically compare explainability techniques, assessing their strengths, weaknesses, and robustness for specific AI tasks.
2. Formulate problems within Pearl's causal hierarchy, constructing structural causal models and causal diagrams to reason about associational, interventional, and counterfactual queries.
3. Apply causal discovery algorithms to infer causal structure from data and evaluate causal decision-making approaches in appropriate problem settings.
4. Employ modern Python libraries for causality and explainable AI to analyse the internal behaviour of machine learning models.
Module syllabus
1. Explainable AI.
- Forms of post-hoc explanation in XAI: attribution-based methods and (non-causal) counterfactuals.
- Robustness of explanations.
- Mechanistic interpretability.
- Explanations for sequential decision-making, including reinforcement learning.
2. Causal reasoning.
- Pearl's causal hierarchy and the foundations of causal reasoning.
- Structural causal models and causal diagrams.
- Causal effects and identifiability.
- The interventional calculus.
- Counterfactual reasoning: definitions, foundations, constraints, and calculus.
3. Causal discovery.
- Inferring causal structure from data.
- Causal decision-making, including causal reinforcement learning.
Teaching methods
The module consists of a combination of lectures, tutorials, and lab sessions. All three parts will have lectures. The part on XAI will have lab sessions in which students implement an ML model and apply modern XAI tools to analyse its behaviour. The part on causal reasoning and the part on learning and decision-making with causality will be supported by tutorials, in which students work through exercises in class as preparation for the written exam. Teaching will be supported by Q&A on an online discussion board, and TAs will assist with tutorials, lab sessions, and marking.
Assessments
The module is assessed by a combination of a written examination (80%) and coursework (20%). The written examination is the primary assessment of theoretical knowledge and, while it covers the breadth of the curriculum, the majority of questions will focus on causality. The coursework consists of a laboratory exercise providing hands-on experience with the explainability of machine learning models, in which students run an interpretability analysis on a model and submit a technical report documenting their methodology, findings, and evaluation of the model's behaviour.
The CW will receive individual feedback. Marks and feedback will be returned within 2 weeks. The examination will receive general (module-wide feedback, in line with departmental practice).