https://arxiv.org/abs/2206.14486
Beyond neural scaling laws: beating power law scaling via data pruning
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, Ari S. Morcos
[Submitted on 29 Jun 2022 (v1), last revised 15 Nov 2022 (this version, v5)]
“Overall, our work suggests that the discovery of good data-pruning metrics may provide a viable path forward to substantially improved neural scaling laws, thereby reducing the resource costs of modern deep learning.”