What I Read: Transformers Training

https://www.borealisai.com/research-blogs/tutorial-17-transformers-iii-training/

Tutorial #17: Transformers III Training
08/06/2021
P. Xu, S. Prince

“…we discuss challenges with transformer training dynamics and introduce some of the tricks that practitioners use to get transformers to converge.”