data engineering – Andrew Fairless, Ph.D.

What I Read: Big Data is Dead

By Andrew Fairless on November 4, 2024August 3, 2024

https://motherduck.com/blog/big-data-is-dead Big Data is DeadJordan Tigani2023/02/07 “Misunderstanding of this point leads to a lot of the discussion of Big Data, because techniques for dealing with large compute requirements are differentContinue readingWhat I Read: Big Data is Dead

What I Read: Data Flywheels, LLM

By Andrew Fairless on October 17, 2024July 20, 2024

https://www.sh-reya.com/blog/ai-engineering-flywheel Data Flywheels for LLM ApplicationsShreya ShankarJul 1, 2024 “This diagram illustrates my (idealized) architecture of an LLM pipeline, from input processing through evaluation and logging. It showcases ideas I’llContinue readingWhat I Read: Data Flywheels, LLM

What I Read: Musings on AI Engineering

By Andrew Fairless on September 16, 2024June 29, 2024

https://www.sh-reya.com/blog/ai-engineering-short Short Musings on AI Engineering and “Failed AI Projects”Shreya ShankarJun 24, 2024 “When we studied the traditional MLOps lifecycle, we found one common pattern across all successful ML products—theirContinue readingWhat I Read: Musings on AI Engineering

What I Read: SSD, Database

By Andrew Fairless on September 12, 2024June 29, 2024

https://cedardb.com/blog/ssd_latency Why Your SSD (Probably) Sucks and What Your Database Can Do About ItLukas VogelJune 18, 2024 “SSDs have effectively replaced spinning disks as the go-to solution for persistent storageContinue readingWhat I Read: SSD, Database

What I Read: Command-line Tools, Faster

By Andrew Fairless on September 10, 2024June 29, 2024

https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html Command-line Tools can be 235x Faster than your Hadoop ClusterAdam Drake “One especially under-used approach for data processing is using standard shell tools and commands. The benefits of thisContinue readingWhat I Read: Command-line Tools, Faster

What I Read: Scaling to Multi-Terabyte Datasets

By Andrew Fairless on September 9, 2024June 29, 2024

Lessons Learned from Scaling to Multi-Terabyte Datasets v2thegreat, AliLessons Learned from Scaling to Multi-Terabyte DatasetsJune 19, 2024 “This post is meant to guide you through some of the lessons I’ve learnedContinue readingWhat I Read: Scaling to Multi-Terabyte Datasets

What I Read: string ufuncs, NumPy 2.0

By Andrew Fairless on August 27, 2024June 15, 2024

https://labs.quansight.org/blog/numpy-string-ufuncs Writing fast string ufuncs for NumPy 2.0Lysandros NikolaouPublished May 21, 2024 “After a huge amount of work from many people, NumPy 2.0.0 will soon be released, the first NumPyContinue readingWhat I Read: string ufuncs, NumPy 2.0

What I Read: PostgreSQL

By Andrew Fairless on July 23, 2024May 24, 2024

https://ottertune.com/blog/the-part-of-postgresql-we-hate-the-most The part of PostgreSQL we hate the mostBohan Zhang | Andy PavloApr 26, 2023 “But as much as we love PostgreSQL at OtterTune, certain aspects of it are notContinue readingWhat I Read: PostgreSQL

What I Read: anti-patterns, data reuse

By Andrew Fairless on July 10, 2024May 6, 2024

https://www.milesmcbain.com/posts/data-analysis-reuse Patterns and anti-patterns of data analysis reuseMiles McBainMarch 10, 2024 “When I look at that function signature I hear the screams of matrices being ground to a fine powderContinue readingWhat I Read: anti-patterns, data reuse

What I Read: predicate pushdown

By Andrew Fairless on May 29, 2024March 23, 2024

https://pola.rs/posts/predicate-pushdown-query-optimizer/ The power of predicate pushdownPolars development teamTue, 19 Mar 2024 “…we will dive into the query optimizer and explain how one of the most important optimization rules works: predicateContinue readingWhat I Read: predicate pushdown

Tag: data engineering