Title
Dynamically Augmented CVaR for Markov Decision Processes
Abstract
This talk describes a new approach to optimization of Conditional Value-at-Risk (CVaR) for Markov Decision Processes (MDPs) with finite state and action sets. It introduces the Dynamically augmented CVaR (DCVaR) risk objective function and provides an algorithm for its optimization. CVaR optimization for MDPs is relevant to a specially defined Robust MDP (RMDP), in which the state space is augmented with the tail risk level. This RMDP, which we call the Dynamically augmented RMDP (DRMDP), was introduced to the literature for calculations of optimal CVaR values by value iteration more than ten years ago, but, as was understood later, these value iterations compute lower bounds of minimal static CVaRs. DCVaR is defined as a time consistent version of the static CVaR, and it is a lower bound of the static CVaR. It also can be viewed as a dynamic version of the nested CVaR. This paper provides an algorithm constructing a policy optimizing DCVaR of total discounted costs. The correctness of this algorithm is proved by studying a special mass transfer problem.
Bio
Eugene A. Feinberg is an American mathematician and distinguished professor of applied mathematics and statistics at Stony Brook University. He is noted for his work in probability theory, real analysis, and Markov decision processes. He completed his PhD thesis at Vilnius University in 1979. Since 1989, he is faculty at the Applied Mathematics and Statistics department of Stony Brook University.