Title: Analytic expressions for the output evolution of a deep neural network during training
Abstract: In this talk we will focus on gaining insight into the output evolution of a deep neural network during training through analytic expressions. We will then use these expressions to understand the effects of the hyperparameters of the optimization algorithm on the output and generalization capabilities of the network. We will start with discussing a previously obtained result which shows that under specific assumptions a deep neural network becomes equivalent to a linear model. In this case one can explicitly solve for the network output during training, and the effects of the training hyperparameters can be studied. For general deep networks the linear approximation is no longer sufficient and higher order approximations are required. Obtaining explicitexpressions in this case if however no longer trivial. We present a Taylor-expansion based method to solve for higher-order approximations of the network output during training, and also in this case study the effects of the hyperparameters of the optimization algorithm on the network output and generalization capabilities.