Speaker:
Prof. Dr. Arnulf Jentzen
Affiliations:
[1] School of Data Science and Shenzhen Research Institut of Big Data, The Chinese University of Hong Kong, Shenzhen, China
[2] Applied Mathematics: Institute of Analysis and Numerics, Faculty of Mathematics and Computer Science, University of Münster, Germany
Title:
Overcoming the curse of dimensionality: from nonlinear Monte Carlo to the training of neural networks
Abstract:
Partial differential equations (PDEs) are among the most universal tools used in modelling problems in nature and man-made complex systems. Nearly all traditional approximation algorithms for PDEs in the literature suffer from the so-called “curse of dimensionality” in the sense that the number of required computational operations of the approximation algorithm to achieve a given approximation accuracy grows exponentially in the dimension of the considered PDE. With such algorithms it is impossible to approximatively compute solutions of high-dimensional PDEs even when the fastest currently available computers are used. In the case of linear parabolic PDEs and approximations at a fixed space-time point, the curse of dimensionality can be overcome by means of Monte Carlo approximation algorithms and the Feynman-Kac formula. In this talk we present an efficient machine learning algorithm to approximate solutions of high-dimensional PDE and we also prove that deep artificial neural network (ANNs) do indeed overcome the curse of dimensionality in the case of a general class of semilinear parabolic PDEs. Moreover, we specify concrete examples of smooth functions which can not be approximated by shallow ANNs without the curse of dimensionality but which can be approximated by deep ANNs without the curse of dimensionality. In the final part of the talk we present some recent mathematical results on the training of neural networks.
References:
[1] Becker, S., Jentzen, A., Müller, M. S., and von Wurstemberger, P., Learning the random variables in Monte Carlo simulations with stochastic gradient descent: Machine learning for parametric PDEs and financial derivative pricing. arXiv:2202.02717 (2022), 70 pages. To appear in Math. Financ.
[2] Beck, C., Becker, S., Cheridito, P., Jentzen, A., and Neufeld, A., Deep splitting method for parabolic PDEs. SIAM J. Sci. Comput. 43 (2021), no. 5, A3135–A3154.
[3] Han, J., Jentzen, A., and E, W., Solving high-dimensional partial differential equations using deep learning. Proc. Natl. Acad. Sci. USA 115 (2018), no. 34, 8505–8510.
[4] Gallon, D., Jentzen, A., Lindner, F., Blow up phenomena for gradient descent optimization methods in the training of artificial neural networks. arXiv:2211.15641 (2023), 84 pages.
[5] Gonon, L., Graeber, R., and Jentzen, A., The necessity of depth for artificial neural networks to approximate certain classes of smooth and bounded functions without the curse of dimensionality. arXiv:2301.08284 (2022), 101 pages.
[6] Hutzenthaler, M., Jentzen, A., Kruse, T., and Nguyen, T. A., A proof that rectified deep neural networks overcome the curse of dimensionality in the numerical approximation of semilinear heat equations. Partial Differ. Equ. Appl. 1 (2020), no. 2, Paper no. 10, 34 pp.
[7] Jentzen, A. and Riekert, A., On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networks. J. Mach. Learn. 1 (2022), no. 2, 141–246.
[8] Ibragimov, S., Jentzen, A., Riekert, A.,
Convergence to good non-optimal critical points in the training of neural networks: Gradient descent optimization with one random initialization overcomes all bad non-global local minima with high probability. arXiv:2212.13111 (2023), 98 pages.