A Stochastic Trust Region Algorithm Based on Careful Step Normalization

References

• Agarwal A, Bottou L (2015) A lower bound for the optimization of finite sums. Proc. Machine Learn. Res. 37:78–86.Google Scholar
• Bottou L, Curtis FE, Nocedal J (2018) Optimization methods for large-scale machine learning. SIAM Rev. 60(2):223–311.Google Scholar
• Byrd RH, Chin GM, Nocedal J, Wu Y (2012) Sample size selection in optimization methods for machine learning. Math. Programming Ser. B 134(1):127–155.Google Scholar
• Chang CC, Lin CJ (2011) LIBSVM: A library for support vector machines. ACM Trans. Intelligent Systems Tech. 2(3):1–27 (Article 27).Google Scholar
• Chen R, Menickelly M, Scheinberg K (2018) Stochastic optimization using a trust-region method and random models. Math. Programming 169(2):447–487.Google Scholar
• Chung KL (1954) On a stochastic approximation method. Ann. Math. Statist. 25(3):463–483.Google Scholar
• Dang C, Lan G (2015) Stochastic block mirror descent methods for nonsmooth and stochastic optimization. SIAM J. Optim. 25(2):856–881.Google Scholar
• Duchi J, Hazan E, Singer Y (2011) Adaptive subgradient methods for online learning and stochastic optimization. J. Machine Learn. Res. 12:2121–2159.Google Scholar
• Freund JE (1962) Mathematical Statistics (Prentice-Hall, Englewood Cliffs, NJ).Google Scholar
• Friedlander MP, Schmidt M (2012) Hybrid deterministic-stochastic methods for data-fitting. SIAM J. Sci. Comput. 34(3):A1380–A1405.Google Scholar
• Ghadimi S, Lan G (2013) Stochastic first- and zeroth-order methods for nonconvex stochastic programming. SIAM J. Optim. 23(4):2341–2368.Google Scholar
• Ghadimi S, Lan G, Zhang H (2016) Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Math. Programming 155(1):267–305.Google Scholar
• Gladyshev EG (1965) On stochastic approximations. Theoret. Probab. Appl. 10(2):275–278.Google Scholar
• Hazan E, Levy K, Shalev-Shwartz S (2015) Beyond convexity: Stochastic quasi-convex optimization. Cortes C, Lawrence ND, Lee DD, Sugiyama M, Garnett R, eds. Advances in Neural Information Processing Systems, vol. 28 (Curran Associates, Red Hook, NY), 1594–1602.Google Scholar
• Jalilzadeh A, Shanbhag UV, Blanchet JH, Glynn PW (2018) Optimal smoothed variable sample-size accelerated proximal methods for structured nonsmooth stochastic convex programs. Working paper, Pennsylvania State University, State College.Google Scholar
• Jofré A, Thompson P (2018) On variance reduction for stochastic smooth convex optimization with multiplicative noise. Math. Programming 174(1–2):253–292.Google Scholar
• Johnson R, Zhang T (2013) Accelerating stochastic gradient descent using predictive variance reduction. Burges CJC, Bottou L, Welling M, Ghahramani Z, Weinberger KQ, eds. Advances in Neural Information Processing Systems, vol. 26 (Curran Associates, Red Hook, NY), 315–323.Google Scholar
• Karimi H, Nutini J, Schmidt M (2016) Linear convergence of gradient and proximal-gradient methods under the Polyak–Łojasiewicz condition. Frasconi P, Landwehr N, Manco G, Vreeken J, eds. Machine Learning and Knowledge Discovery in Databases, Lecture Notes in Computer Science, vol. 9851 (Springer, Cham, Switzerland), 795–811.Google Scholar
• Krizhevsky A (2009) Learning multiple layers of features from tiny images. Technical report, University of Toronto, Toronto.Google Scholar
• Larson J, Billups SC (2016) Stochastic derivative-free optimization using a trust region framework. Comput. Optim. Appl. 64(3):619–645.Google Scholar
• LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc. IEEE 86(11):2278–2324.Google Scholar
• Lei J, Shanbhag UV (2018) A randomized block proximal variable sample-size stochastic gradient method for composite nonconvex stochastic optimization. Working paper, Pennsylvania State University, State College.Google Scholar
• Łojasiewicz S (1963) Une propriété topologique des sous ensembles analytiques réels. Les Équations aux Dérivées Partielles, Colloques Internationaux du C.N.R.S 117, 87–89.Google Scholar
• Nemirovski A, Juditsky A, Lan G, Shapiro A (2009) Robust stochastic approximation approach to stochastic programming. SIAM J. Optim. 19(4):1574–1609.Google Scholar
• Polyak BT (1963) Gradient methods for minimizing functionals. Zh. Vychisl. Mat. Mat. Fiz. 3:643–653.Google Scholar
• Robbins H, Monro S (1951) A stochastic approximation method. Ann. Math. Statist. 22(3):400–407.Google Scholar
• Robbins H, Siegmund D (1971) A convergence theorem for nonnegative almost supermartingales and some applications. Rustagi JS, ed. Optimizing Methods in Statistics (Academic Press, Cambridge, MA).Google Scholar
• Ross S, Mineiro P, Langford J (2013) Normalized online learning. Proc. 29th Conf. Uncertainty Artificial Intelligence, Bellevue, WA, 537–545.Google Scholar
• Shapiro A, Dentcheva D, Ruszczyński A (2009) Lectures on Stochastic Programming: Modeling and Theory (Society for Industrial and Applied Mathematics, Philadelphia).Google Scholar
• Tong YL (2012) The Multivariate Normal Distribution (Springer Science+Business Media, Berlin).Google Scholar
• Xie Y, Shanbhag UV (2016) SI-ADMM: A stochastic inexact ADMM framework for resolving structured stochastic convex programs. Proc. 2016 Winter Simulation Conf. (IEEE, Piscataway, NJ), 714–725.Google Scholar
• Xie Y, Shanbhag UV (2017) SI-ADMM: A stochastic inexact ADMM framework for resolving structured stochastic convex programs. Working paper, Pennsylvania State University, State College.Google Scholar
• Xu Y, Yin W (2015) Block stochastic gradient iteration for convex and nonconvex optimization. SIAM J. Optim. 25(3):1686–1716.Google Scholar