http://www.offconvex.org/2020/10/21/intrinsicLR/
Mismatches between Traditional Optimization Analyses and Modern Deep Learning
Zhiyuan Li and Sanjeev Arora
Oct 21, 2020
“You may remember our previous blog post showing that it is possible to do state-of-the-art deep learning with learning rate that increases exponentially during training. It was meant to be a dramatic illustration that what we learned in optimization classes and books isn’t always a good fit for modern deep learning… Today’s post… identifies other surprising incompatibilities… We hope this will change the way you teach and think about deep learning!”