Skip to main content

An improved convergence analysis of cyclic block coordinate descent-type methods for strongly convex minimization

Author(s): Li, X; Zhao, T; Arora, R; Liu, H; Hong, M

Download
To refer to this page use: http://arks.princeton.edu/ark:/88435/pr1000q
Abstract: The cyclic block coordinate descent-type (CBCD-type) methods have shown remarkable computational performance for solving strongly convex minimization problems. Typical applications include many popular statistical machine learning methods such as elastic-net regression, ridge penalized logistic regression, and sparse additive regression. Existing optimization literature has shown that the CBCD-type methods attain iteration complexity of O(p⋅\log(1/ε)), where εis a pre-specified accuracy of the objective value, and p is the number of blocks. However, such iteration complexity explicitly depends on p, and therefore is at least p times worse than those of gradient descent methods. To bridge this theoretical gap, we propose an improved convergence analysis for the CBCD-type methods. In particular, we first show that for a family of quadratic minimization problems, the iteration complexity of the CBCD-type methods matches that of the GD methods in term of dependency on p (up to a \log^2 p factor). Thus our complexity bounds are sharper than the existing bounds by at least a factor of p/\log^2p. We also provide a lower bound to confirm that our improved complexity bounds are tight (up to a \log^2 p factor) if the largest and smallest eigenvalues of the Hessian matrix do not scale with p. Finally, we generalize our analysis to other strongly convex minimization problems beyond quadratic ones.
Publication Date: 2016
Citation: Li, Xingguo, Tuo Zhao, Raman Arora, Han Liu, and Mingyi Hong. "An improved convergence analysis of cyclic block coordinate descent-type methods for strongly convex minimization." Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, PMLR, 51 (2016): 491-499.
Pages: 491 - 499
Type of Material: Conference Article
Series/Report no.: Proceedings of Machine Learning Research;
Journal/Proceeding Title: Proceedings of the 19th International Conference on Artificial Intelligence and Statistics
Version: Final published version. Article is made available in OAR by the publisher's permission or policy.



Items in OAR@Princeton are protected by copyright, with all rights reserved, unless otherwise indicated.