https://distill.pub/2020/circuits/branch-specialization/
Branch Specialization
Chelsea Voss
Gabriel Goh
Nick Cammarata
Michael Petrov
Ludwig Schubert
Chris Olah
April 5, 2021
DOI 10.23915/distill.00024.008
“Branch specialization occurs when neural network layers are split up into branches. The neurons and circuits tend to self-organize, clumping related functions into each branch and forming larger functional units – a kind of “neural network brain region.” We find evidence that these structures implicitly exist in neural networks without branches, and that branches are simply reifying structures that otherwise exist.”