data engineering – Andrew Fairless, Ph.D.

What I Read: data migrations

By Andrew Fairless on February 19, 2025November 9, 2024

https://yorickpeterse.com/articles/building-a-better-and-scalable-system-for-data-migrations Building a better and scalable system for data migrationsYorick PeterseOctober 24, 2024 “I’ve been thinking about what a better solution to data migrations might look like for a while,Continue readingWhat I Read: data migrations

What I Read: Big Data is Dead

By Andrew Fairless on November 4, 2024August 3, 2024

https://motherduck.com/blog/big-data-is-dead Big Data is DeadJordan Tigani2023/02/07 “Misunderstanding of this point leads to a lot of the discussion of Big Data, because techniques for dealing with large compute requirements are differentContinue readingWhat I Read: Big Data is Dead

What I Read: Data Flywheels, LLM

By Andrew Fairless on October 17, 2024July 20, 2024

https://www.sh-reya.com/blog/ai-engineering-flywheel Data Flywheels for LLM ApplicationsShreya ShankarJul 1, 2024 “This diagram illustrates my (idealized) architecture of an LLM pipeline, from input processing through evaluation and logging. It showcases ideas I’llContinue readingWhat I Read: Data Flywheels, LLM

What I Read: Musings on AI Engineering

By Andrew Fairless on September 16, 2024June 29, 2024

https://www.sh-reya.com/blog/ai-engineering-short Short Musings on AI Engineering and “Failed AI Projects”Shreya ShankarJun 24, 2024 “When we studied the traditional MLOps lifecycle, we found one common pattern across all successful ML products—theirContinue readingWhat I Read: Musings on AI Engineering

What I Read: SSD, Database

By Andrew Fairless on September 12, 2024June 29, 2024

https://cedardb.com/blog/ssd_latency Why Your SSD (Probably) Sucks and What Your Database Can Do About ItLukas VogelJune 18, 2024 “SSDs have effectively replaced spinning disks as the go-to solution for persistent storageContinue readingWhat I Read: SSD, Database

What I Read: Command-line Tools, Faster

By Andrew Fairless on September 10, 2024June 29, 2024

https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html Command-line Tools can be 235x Faster than your Hadoop ClusterAdam Drake “One especially under-used approach for data processing is using standard shell tools and commands. The benefits of thisContinue readingWhat I Read: Command-line Tools, Faster

What I Read: Scaling to Multi-Terabyte Datasets

By Andrew Fairless on September 9, 2024June 29, 2024

Lessons Learned from Scaling to Multi-Terabyte Datasets v2thegreat, AliLessons Learned from Scaling to Multi-Terabyte DatasetsJune 19, 2024 “This post is meant to guide you through some of the lessons I’ve learnedContinue readingWhat I Read: Scaling to Multi-Terabyte Datasets

What I Read: string ufuncs, NumPy 2.0

By Andrew Fairless on August 27, 2024June 15, 2024

https://labs.quansight.org/blog/numpy-string-ufuncs Writing fast string ufuncs for NumPy 2.0Lysandros NikolaouPublished May 21, 2024 “After a huge amount of work from many people, NumPy 2.0.0 will soon be released, the first NumPyContinue readingWhat I Read: string ufuncs, NumPy 2.0

Tag: data engineering