Skip to main content

An Adaptive Learning Rate for Stochastic Variational Inference

Author(s): Ranganath, Rajesh; Wang, Chong; Blei, David M; Xing, Eric P

To refer to this page use:
Abstract: Stochastic variational inference finds good posterior approximations of probabilistic models with very large data sets. It optimizes the variational objective with stochastic optimization, following noisy estimates of the natural gradient. Operationally, stochastic inference iteratively subsamples from the data, analyzes the subsample, and updates parameters with a decreasing learning rate. However, the algorithm is sensitive to that rate, which usually requires hand-tuning to each application. We solve this problem by developing an adaptive learning rate for stochastic inference. Our method requires no tuning and is easily implemented with computations already made in the algorithm. We demonstrate our approach with latent Dirichlet allocation applied to three large text corpora. Inference with the adaptive learning rate converges faster and to a better approximation than the best settings of hand-tuned rates.
Publication Date: 2013
Citation: Ranganath, Rajesh, Chong Wang, Blei David, and Eric Xing. "An Adaptive Learning Rate for Stochastic Variational Inference." In International Conference on Machine Learning (2013): pp. 298-306.
Pages: 298 - 306
Type of Material: Conference Article
Journal/Proceeding Title: International Conference on Machine Learning
Version: Final published version. Article is made available in OAR by the publisher's permission or policy.

Items in OAR@Princeton are protected by copyright, with all rights reserved, unless otherwise indicated.