PhD Candidate, Cornell University
Email
Abstract
Snapshot Ensemble: Train 1, Get M for Free
Ensembles of neural networks are known to be much more robust and accurate than individual networks. However, training multiple deep networks for model averaging is computationally expensive. In this paper, we propose a method to obtain the seemingly contradictory goal of ensembling multiple neural networks at no additional training cost. We achieve this goal by training a single neural network, converging to several local minima along its optimization path and saving the model parameters. To obtain repeated rapid convergence, we leverage recent work on cyclic learning rate schedules. The resulting technique, which we refer to as Snapshot Ensembling, is simple, yet surprisingly effective. We show in a series of experiments that our approach is compatible with diverse network architectures and learning tasks. It consistently yields lower error rates than state-of-the-art single models at no additional training cost, and compares favorably with traditional network ensembles. On CIFAR-10 and CIFAR-100 our DenseNet Snapshot Ensembles obtain error rates of 3.4% and 17.4% respectively.
Bio
Yixuan Li is a Ph.D. candidate at Cornell University, advised by Professor John Hopcroft. Her thesis committee members are Kilian Weinberger and Thorsten Joachims. Her research focuses on both the theoretical and applied aspects of machine learning and perception. Yixuan is particularly interested in large-scale machine learning for the web, with topics including scalable semi-supervised learning, deep representation learning for vision tasks, user modeling in social media, and visual attention based personalization etc. A key focus of her recent work has been on deep learning. Projects include convergent learning in deep neural networks, optimizing neural networks with efficient computational cost, adversarial training of deep generative models, improving neural network safety, and theorectical aspect of deep learning. Yixuan has published in leading machine learning and computer vision conferences, such as NIPS, ICLR, CVPR and WWW. Prior to Cornell, she graduated from Shanghai Jiaotong University with B.Eng in Information Engineering in 2013. She was twice a research intern with Google Research (Mountain View) in 2015 and 2016.