training – Page 2 – Andrew Fairless, Ph.D.

What I Read: Finetuning LLMs Using LoRA

By Andrew Fairless on February 5, 2024December 18, 2023

https://magazine.sebastianraschka.com/p/practical-tips-for-finetuning-llms Practical Tips for Finetuning LLMs Using LoRA (Low-Rank Adaptation)Sebastian Raschka, PhDNov 19, 2023 “Low-rank adaptation (LoRA) is among the most widely used and effective techniques for efficiently training customContinue readingWhat I Read: Finetuning LLMs Using LoRA

What I Read: Gaussian Processes Extrapolate

By Andrew Fairless on January 29, 2024November 20, 2023

https://probablymarcus.com/blocks/2023/03/28/gp-extrapolation-sometimes-goofy.html Gaussian Processes Extrapolate, Sometimes in Goofy WaysMarcus Lewis3/28/23 4:59 AM “This leads to something that is undesirable in hyperparameter tuning: if ever you discover a sudden drop-off in performance…Continue readingWhat I Read: Gaussian Processes Extrapolate

What I Read: Unify Batch and ML Systems

By Andrew Fairless on January 18, 2024November 7, 2023

https://www.kdnuggets.com/2023/09/hopsworks-unify-batch-ml-systems-feature-training-inference-pipelines Unify Batch and ML Systems with Feature/Training/Inference PipelinesBy Jim Dowling, Co-Founder & CEO, HopsworksSeptember 27, 2023 “This article introduces a unified architectural pattern for building both Batch and Real-TimeContinue readingWhat I Read: Unify Batch and ML Systems

What I Read: Finetuning LLMs with LoRA and QLoRA

By Andrew Fairless on January 3, 2024November 7, 2023

Finetuning LLMs with LoRA and QLoRA: Insights from Hundreds of Experiments Finetuning LLMs with LoRA and QLoRA: Insights from Hundreds of ExperimentsSebastian RaschkaOctober 12, 2023 “LoRA is one of theContinue readingWhat I Read: Finetuning LLMs with LoRA and QLoRA

What I Read: Auditing AI, How Much Access

By Andrew Fairless on November 14, 2023October 5, 2023

https://aipolicy.substack.com/p/ai-accountability-transparency-2 Auditing AI: How Much Access Is Needed to Audit an AI System?Chapter 2 in our AI Accountability & Transparency series(By Sarah H. Cen, Cosimo L. Fabrizio, James Siderius, AleksanderContinue readingWhat I Read: Auditing AI, How Much Access

What I Read: Nvidia AI Supremacy Temporary

By Andrew Fairless on October 25, 2023October 5, 2023

https://petewarden.com/2023/09/10/why-nvidias-ai-supremacy-is-only-temporary/ Why Nvidia’s AI Supremacy is Only TemporarySeptember 10, 2023Pete Warden “…hopefully I’ve made a convincing case that there are strong structural reasons behind Nvidia’s success. Here’s how I seeContinue readingWhat I Read: Nvidia AI Supremacy Temporary

What I Read: LLMs, single example

By Andrew Fairless on October 23, 2023September 9, 2023

https://www.fast.ai/posts/2023-09-04-learning-jumps/ Can LLMs learn from a single example?Jeremy Howard and Jonathan WhitakerSeptember 4, 2023 “We’ve noticed an unusual training pattern in fine-tuning LLMs. At first we thought it’s a bug,Continue readingWhat I Read: LLMs, single example

What I Read: Models Memorize or Generalize?

By Andrew Fairless on October 2, 2023September 4, 2023

https://pair.withgoogle.com/explorables/grokking/ Do Machine Learning Models Memorize or Generalize?By Adam Pearce, Asma Ghandeharioun, Nada Hussein, Nithum Thain, Martin Wattenberg and Lucas DixonAugust 2023 “Large language models can certainly seem like theyContinue readingWhat I Read: Models Memorize or Generalize?

What I Read: scaling laws, cross-entropy loss

By Andrew Fairless on September 28, 2023September 4, 2023

https://epochai.org/blog/the-direct-approach The Direct ApproachMatthew Barnett and Tamay Besiroglu “Empirical scaling laws can help predict the cross-entropy loss associated with training inputs, such as compute and data. However, in order toContinue readingWhat I Read: scaling laws, cross-entropy loss

What I Read: Attention Off By One

By Andrew Fairless on September 13, 2023September 4, 2023

https://www.evanmiller.org/attention-is-off-by-one.html Attention Is Off By OneBy Evan MillerJuly 24, 2023 “…the current generation of AI models have an off-by-one error in a crucial place, and it’s making everyone’s Transformer modelsContinue readingWhat I Read: Attention Off By One

Tag: training