https://www.kdnuggets.com/2020/03/deep-learning-breakthrough-sub-linear-algorithm-no-gpu.html
Deep Learning Breakthrough: a sub-linear deep learning algorithm that does not need a GPU?
By Anshumali Shrivastava, Rice University.
“Backpropagation is not an efficient algorithm…. The core idea behind the SLIDE algorithm is to minimize wasteful computations in backpropagation…. Backpropagation is oblivious to adaptive sparsity. Instead, backpropagation updates millions of parameters using full matrix multiplication. Most of these updates are unnecessary.”