scalability – Page 2 – Andrew Fairless, Ph.D.

What I Read: smaller LLMs, more tokens

By Andrew Fairless on May 31, 2023May 1, 2023

https://www.harmdevries.com/post/model-size-vs-compute-overhead/ Go smol or go homeWhy we should train smaller LLMs on more tokensHarm de VriesApr 13, 2023 “However, for most use cases you should not train a compute-optimal LLMContinue readingWhat I Read: smaller LLMs, more tokens

What I Read: databases

By Andrew Fairless on May 2, 2023March 30, 2023

https://www.thenile.dev/blog/things-dbs-dont-do Things DBs Don’t Do – But ShouldFebruary 6, 2023Gwen Shapira “Why do I still need to implement this in every project? why doesn’t the DB just take care ofContinue readingWhat I Read: databases

What I Read: Bloom filter

By Andrew Fairless on April 26, 2023March 30, 2023

https://luminousmen.com/post/building-a-bloom-filter Building a Bloom filterluminousmen “In this post, we will explore the Bloom filter — a data structure that is ingenious in its simplicity and elegant in its design.”

What I Learn: neural scaling, data pruning

By Andrew Fairless on April 11, 2023February 22, 2023

https://arxiv.org/abs/2206.14486 Beyond neural scaling laws: beating power law scaling via data pruningBen Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, Ari S. Morcos[Submitted on 29 Jun 2022 (v1), last revised 15Continue readingWhat I Learn: neural scaling, data pruning

What I Read: Key-Value Databases

By Andrew Fairless on December 5, 2022October 20, 2022

https://www.kdnuggets.com/2021/04/nosql-explained-understanding-key-value-databases.html Key-Value Databases, ExplainedBy Alex Williams, Hosting Data UK on October 4, 2022 in SQL “Among the four big NoSQL database types, key-value stores are probably the most popular onesContinue readingWhat I Read: Key-Value Databases

What I Read: Emergent Features

By Andrew Fairless on October 6, 2022August 22, 2022

LLM.int8() and Emergent Features LLM.int8() and Emergent Features2022-08-17 by Tim Dettmers “…I use advanced quantization methods to achieve no performance degradation transformer inference at scale that makes large models moreContinue readingWhat I Read: Emergent Features

What I Read: Against Naive AI Scaling

By Andrew Fairless on July 27, 2022June 27, 2022

https://jacobbuckman.com/2022-06-14-an-actually-good-argument-against-naive-ai-scaling/ An Actually-Good Argument Against Naive AI ScalingJacob BuckmanPosted on June 14, 2022 “…the debate is whether scaled-up language models in the style of GPT-3 will eventually become general intelligences,Continue readingWhat I Read: Against Naive AI Scaling

What I Watch: Engineering For Data

By Andrew Fairless on March 16, 2022January 20, 2022

https://youtu.be/jeMQld-6BtQ What Is Engineering For Data?Jan 5, 2022Continuous Delivery “What are the big mistakes that data scientists and big organisations often make when dealing with big data? What does engineeringContinue readingWhat I Watch: Engineering For Data

What I Read: How Train Large Deep Learning Models

By Andrew Fairless on November 8, 2021October 24, 2021

https://www.assemblyai.com/blog/how-to-train-large-deep-learning-models-as-a-startup/ How to Train Large Deep Learning Models as a StartupDylan FoxOct 7, 2021Training large deep learning models is expensive and slow. Yet, startups are all about iterating fast. InContinue readingWhat I Read: How Train Large Deep Learning Models

What I Read: How to Train Really Large Models

By Andrew Fairless on October 26, 2021October 16, 2021

https://lilianweng.github.io/lil-log/2021/09/24/train-large-neural-networks.html How to Train Really Large Models on Many GPUs?Sep 24, 2021Lilian Weng “How to train large and deep neural networks is challenging, as it demands a large amount ofContinue readingWhat I Read: How to Train Really Large Models

Tag: scalability