Skip to main content

Efficient optimization of loops and limits with randomized telescoping sums

Author(s): Beatson, Alex; Adams, Ryan P

To refer to this page use:
Abstract: We consider optimization problems in which the objective requires an inner loop with many steps or is the limit of a sequence of increasingly costly approximations. Meta-learning, training recurrent neural networks, and optimization of the solutions to differential equations are all examples of optimization problems with this character. In such problems, it can be expensive to compute the objective function value and its gradient, but truncating the loop or using less accurate approximations can induce biases that damage the overall solution. We propose randomized telescope (RT) gradient estimators, which represent the objective as the sum of a telescoping series and sample linear combinations of terms to provide cheap unbiased gradient estimates. We identify conditions under which RT estimators achieve optimization convergence rates independent of the length of the loop or the required accuracy of the approximation. We also derive a method for tuning RT estimators online to maximize a lower bound on the expected decrease in loss per unit of computation. We evaluate our adaptive RT estimators on a range of applications including meta-optimization of learning rates, variational inference of ODE parameters, and training an LSTM to model long sequences.
Publication Date: 2019
Citation: Beatson, Alex, and Ryan P. Adams. "Efficient optimization of loops and limits with randomized telescoping sums." Proceedings of the 36th International Conference on Machine Learning 97 (2019), pp. 534-543.
ISSN: 2640-3498
Pages: 534 - 543
Type of Material: Conference Article
Journal/Proceeding Title: Proceedings of the 36th International Conference on Machine Learning
Version: Final published version. Article is made available in OAR by the publisher's permission or policy.

Items in OAR@Princeton are protected by copyright, with all rights reserved, unless otherwise indicated.