https://distill.pub/2020/circuits/weight-banding/
Weight Banding
Michael Petrov
Chelsea Voss
Ludwig Schubert
Nick Cammarata
Gabriel Goh
Chris Olah
April 8, 2021
DOI 10.23915/distill.00024.009
“Open up any ImageNet conv net and look at the weights in the last layer. You’ll find a uniform spatial pattern to them, dramatically unlike anything we see elsewhere in the network… the uniformity is so striking that when we first discovered it we thought it must be a bug…. We call this phenomenon weight banding.”