efficiency – Page 2 – Andrew Fairless, Ph.D.

What I Read: Scaling ChatGPT, Engineering Challenges

By Andrew Fairless on April 18, 2024March 1, 2024

https://newsletter.pragmaticengineer.com/p/scaling-chatgpt Scaling ChatGPT: Five Real-World Engineering ChallengesGergely OroszFeb 20, 2024 “Just one year after its launch, ChatGPT had more than 100M weekly users. In order to meet this explosive demand,Continue readingWhat I Read: Scaling ChatGPT, Engineering Challenges

What I Read: Research Directions

By Andrew Fairless on February 21, 2024January 7, 2024

https://nlpnewsletter.substack.com/p/nlp-research-in-the-era-of-llms NLP Research in the Era of LLMs5 Key Research Directions Without Much ComputeSebastian RuderDec 19, 2023 “In an era where running state-of-the-art models requires a garrison of expensive GPUs,Continue readingWhat I Read: Research Directions

What I Read: vectorize wide PyTorch expressions

By Andrew Fairless on January 16, 2024November 7, 2023

https://probablymarcus.com/blocks/2023/10/19/vectorizing-wide-pytorch-expressions.html What happens when you vectorize wide PyTorch expressions?Marcus Lewis19 October 2023 “So we see that vectorization has three effects:It lets us keep the GPU busy even when inputs areContinue readingWhat I Read: vectorize wide PyTorch expressions

What I Read: Compiling NumPy

By Andrew Fairless on January 10, 2024November 7, 2023

https://pytorch.org/blog/compiling-numpy-code/ Compiling NumPy code into C++ or CUDA via torch.compileby Evgeni Burovski, Ralf Gommers and Mario LezcanoOctober 17, 2023 “This feature leverages PyTorch’s compiler to generate efficient fused vectorized codeContinue readingWhat I Read: Compiling NumPy

What I Read: Markov Chains

By Andrew Fairless on October 24, 2023September 9, 2023

https://elevanth.org/blog/2017/11/28/build-a-better-markov-chain/ Markov Chains: Why Walk When You Can Flow?Richard McElreath28 November 2017 “If you are still using a Gibbs sampler, you are working too hard for too little result. Newer,Continue readingWhat I Read: Markov Chains

What I Read: speed up Numba and NumPy

By Andrew Fairless on August 1, 2023June 15, 2023

https://pythonspeed.com/articles/speeding-up-numba/ Understanding CPUs can help speed up Numba and NumPy codeby Itamar Turner-TrauringLast updated 08 Jun 2023, originally created 07 Jun 2023 “In practice, your initial Numba code may beContinue readingWhat I Read: speed up Numba and NumPy

What I Read: smaller LLMs, more tokens

By Andrew Fairless on May 31, 2023May 1, 2023

https://www.harmdevries.com/post/model-size-vs-compute-overhead/ Go smol or go homeWhy we should train smaller LLMs on more tokensHarm de VriesApr 13, 2023 “However, for most use cases you should not train a compute-optimal LLMContinue readingWhat I Read: smaller LLMs, more tokens

What I Read: Functional, Object Oriented Programming

By Andrew Fairless on May 1, 2023March 30, 2023

Functional Programming IS NO BETTER Object Oriented ProgrammingContinuous Delivery “Which of these software paradigms will offer you the best chance of success in which instances?”

What I Learn: neural scaling, data pruning

By Andrew Fairless on April 11, 2023February 22, 2023

https://arxiv.org/abs/2206.14486 Beyond neural scaling laws: beating power law scaling via data pruningBen Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, Ari S. Morcos[Submitted on 29 Jun 2022 (v1), last revised 15Continue readingWhat I Learn: neural scaling, data pruning

What I Read: Optimizing Machine Learning Training Pipelines

By Andrew Fairless on March 15, 2023February 4, 2023

https://medium.com/ntropy-network/bag-of-tricks-for-optimizing-machine-learning-training-pipelines-4f8d5cd3d432 Bag of tricks for optimizing machine learning training pipelinesArseny KravchenkoJan 5 “…we are constantly looking for ways to improve the efficiency of our machine learning pipelines, while keeping theContinue readingWhat I Read: Optimizing Machine Learning Training Pipelines

Tag: efficiency