https://coconut-mode.com/posts/ring-attention
Ring Attention Explained
Kilian Haefeli, Simon Zirui Guo, Bonnie Li
10 Apr 2024
“Context length in Large Language Models has expanded rapidly…. What if we we could use multiple devices to scale to a near infinite context window? Ring Attention is a promising approach to do so…”