Title: Learning from data via overparameterization
Abstract: Solving data-driven problems requires defining complex models and fitting them on data, neural networks being a motivating example. The fitting procedure can be seen as an optimization problem, which is often non-convex, and hence optimization guarantees are hard to derive. An opportunity is provided by viewing the model of interest as a redundant reparameterization—an overparameterization—of some simpler model for which optimization results are easier to achieve. In this talk, after formalizing the above idea, we review some recent results and derive new ones. In particular, we consider the gradient flow of some classes of linear overparameterization and show they correspond to suitable mirror flow on the original parameters. Our main contribution relates to the study of the latter, for which we establish well-posedness and convergence. The results yield insight on the role of overparameterization for implicit regularization and constrained optimization.