https://magazine.sebastianraschka.com/p/practical-tips-for-finetuning-llms Practical Tips for Finetuning LLMs Using LoRA (Low-Rank Adaptation)Sebastian Raschka, PhDNov 19, 2023 “Low-rank adaptation (LoRA) is among the most widely used and effective techniques for efficiently training custom
What I Read: Nvidia, GPU gold rush
https://blog.johnluttig.com/p/nvidia-envy-understanding-the-gpu Nvidia Envy: understanding the GPU gold rushJohn LuttigNov 10, 2023 “In 2023, thousands of companies and countries begged Nvidia to purchase more GPUs. Can the exponential demand endure?”
What I Read: Distributed Training, Finetuning
https://sumanthrh.com/post/distributed-and-efficient-finetuning/ Everything about Distributed Training and Efficient FinetuningSumanth R HegdeLast updated on Oct 13, 2023 “practical guidelines and gotchas with multi-GPU and multi-node training”
What I Read: Tiny Language Models
https://www.quantamagazine.org/tiny-language-models-thrive-with-gpt-4-as-a-teacher-20231005/ Tiny Language Models Come of AgeBen Brubaker10/5/23 10:50 AM “To better understand how neural networks learn to simulate writing, researchers trained simpler versions on synthetic children’s stories.”