parallelization – Page 2 – Andrew Fairless, Ph.D.

What I Read: GPU Computing

By Andrew Fairless on January 15, 2024November 7, 2023

https://codeconfessions.substack.com/p/gpu-computing What Every Developer Should Know About GPU ComputingAbhinav UpadhyayOct 18, 2023 “GPUs have become incredibly important because of their pervasive use in deep learning. Today, it is essential forContinue readingWhat I Read: GPU Computing

What I Read: Retrieval Augmented Generation at scale

By Andrew Fairless on December 13, 2023November 7, 2023

https://medium.com/@neum_ai/retrieval-augmented-generation-at-scale-building-a-distributed-system-for-synchronizing-and-eaa29162521 Retrieval Augmented Generation at scale — Building a distributed system for synchronizing and ingesting billions of text embeddingsNeum AISep 28 “…getting a Retrieval Augmented Generation (RAG) application started isContinue readingWhat I Read: Retrieval Augmented Generation at scale

What I Read: Accelerating PyTorch

By Andrew Fairless on September 6, 2023August 1, 2023

https://magazine.sebastianraschka.com/p/accelerating-pytorch-model-training Accelerating PyTorch Model TrainingUsing Mixed-Precision and Fully Sharded Data ParallelismSebastian Raschka, PhDJun 26, 2023 “…how to scale PyTorch model training with minimal code changes. The focus here is onContinue readingWhat I Read: Accelerating PyTorch

What I Read: How to Train Really Large Models

By Andrew Fairless on October 26, 2021October 16, 2021

https://lilianweng.github.io/lil-log/2021/09/24/train-large-neural-networks.html How to Train Really Large Models on Many GPUs?Sep 24, 2021Lilian Weng “How to train large and deep neural networks is challenging, as it demands a large amount ofContinue readingWhat I Read: How to Train Really Large Models

What I Read: Systems for Machine Learning

By Andrew Fairless on September 14, 2021August 20, 2021

https://thegradient.pub/systems-for-machine-learning/ Systems for Machine LearningKabir Nagrecha14.Aug.2021 “Machine learning’s increasing importance to real-world applications brought awareness of a new field focused on ML in practice – machine learning systems (or, asContinue readingWhat I Read: Systems for Machine Learning

What I Read: Parallel Bayesian Optimization

By Andrew Fairless on May 24, 2021May 1, 2021

https://medium.com/riskified-technology/from-sequential-to-parallel-a-story-about-parallel-bayesian-hyperparameter-optimization-f124ed29f556 From Sequential to Parallel, a story about Bayesian Hyperparameter OptimizationAndres AsaraviciusApr 26 “The goal of this process is to find the best set of parameters, and our goal wasContinue readingWhat I Read: Parallel Bayesian Optimization

What I Read: Weight Banding

By Andrew Fairless on May 5, 2021April 10, 2021

https://distill.pub/2020/circuits/weight-banding/ Weight BandingMichael PetrovChelsea VossLudwig SchubertNick CammarataGabriel GohChris OlahApril 8, 2021DOI 10.23915/distill.00024.009 “Open up any ImageNet conv net and look at the weights in the last layer. You’ll find aContinue readingWhat I Read: Weight Banding

What I Read: Python Concurrency

By Andrew Fairless on March 25, 2021March 7, 2021

https://python.hamel.dev/concurrency/ Python NotesPython Concurrency: The Tricky Bits2020-02-05 concurrency An exploration of threads, processes, and coroutines in Python, with interesting examples that illuminate the differences between each.“I’ve found many data scientistsContinue readingWhat I Read: Python Concurrency

What I Read: Frameworks Scaling Deep Learning Training

By Andrew Fairless on February 9, 2021January 12, 2021

https://medium.com/dataseries/microsoft-and-google-open-sourced-these-frameworks-based-on-their-work-scaling-deep-learning-c0510e907038 Microsoft and Google Open Sourced These Frameworks Based on Their Work Scaling Deep Learning TrainingGoogle and Microsoft have recently released new frameworks for distributed deep learning training.Jesus RodriguezOct 26Continue readingWhat I Read: Frameworks Scaling Deep Learning Training

Tag: parallelization