training – Andrew Fairless, Ph.D.

What I Read: Age of Data

By Andrew Fairless on April 14, 2025January 5, 2025

https://amatria.in/blog/ageofdata The end of the “Age of Data”? Enter the age of superhuman data and AIXavier AmatriainDecember 24, 2024 “This post will argue that the age of data is farContinue readingWhat I Read: Age of Data

What I Read: benchmark

By Andrew Fairless on February 26, 2025November 9, 2024

Run a benchmark they said. It will be fun they said. | PyData Amsterdam 2024Vincent D. WarmerdamOct 22, 2024 “This is the story of a fun idea that turned intoContinue readingWhat I Read: benchmark

What I Read: Toy Models of Superposition

By Andrew Fairless on December 19, 2024September 29, 2024

https://transformer-circuits.pub/2022/toy_model/index.html Toy Models of SuperpositionNelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan,Continue readingWhat I Read: Toy Models of Superposition

What I Read: Visual Guide, Quantization

By Andrew Fairless on October 30, 2024August 3, 2024

https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-quantization A Visual Guide to Quantization, Demystifying the Compression of Large Language ModelsMaarten GrootendorstJul 22, 2024 “…I will introduce the field of quantization in the context of language modeling andContinue readingWhat I Read: Visual Guide, Quantization

What I Read: bare metal to 70B

By Andrew Fairless on September 25, 2024July 8, 2024

https://imbue.com/research/70b-infrastructure From bare metal to a 70B model: infrastructure set-up and scriptsThe Imbue TeamJune 25, 2024 “…we trained a 70B parameter model from scratch on our own infrastructure that outperformedContinue readingWhat I Read: bare metal to 70B

What I Read: How Machines ‘Grok’ Data

By Andrew Fairless on June 25, 2024April 23, 2024

https://www.quantamagazine.org/how-do-machines-grok-data-20240412 How Do Machines ‘Grok’ Data?Anil Ananthaswamy4/12/24 “By apparently overtraining them, researchers have seen neural networks discover novel solutions to problems.”

What I Read: Data Selection, LLMs

By Andrew Fairless on June 12, 2024April 15, 2024

https://www.cs.princeton.edu/~smalladi/blog/2024/04/04/dataselection Using LESS Data to Tune Models: Data Selection in the Era of LLMsMengzhou Xia and Sadhika MalladiApril 04 2024 “We describe how data selection for modern-day LLMs differs fromContinue readingWhat I Read: Data Selection, LLMs

What I Read: Diffusion Model theory

By Andrew Fairless on April 23, 2024March 1, 2024

https://ayandas.me/2024/blog/diffusion-theory-from-scratch/ Building Diffusion Model’s theory from ground upAyan DasMay 7, 2024 “…we will go back and revisit the ‘fundamental ingredients’ behind the SDE formulation, and show how the idea canContinue readingWhat I Read: Diffusion Model theory

What I Read: Scaling ChatGPT, Engineering Challenges

By Andrew Fairless on April 18, 2024March 1, 2024

https://newsletter.pragmaticengineer.com/p/scaling-chatgpt Scaling ChatGPT: Five Real-World Engineering ChallengesGergely OroszFeb 20, 2024 “Just one year after its launch, ChatGPT had more than 100M weekly users. In order to meet this explosive demand,Continue readingWhat I Read: Scaling ChatGPT, Engineering Challenges

Tag: training