data engineering – Page 2 – Andrew Fairless, Ph.D.

What I Read: Command-line Tools, Faster

By Andrew Fairless on September 10, 2024June 29, 2024

https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html Command-line Tools can be 235x Faster than your Hadoop ClusterAdam Drake “One especially under-used approach for data processing is using standard shell tools and commands. The benefits of thisContinue readingWhat I Read: Command-line Tools, Faster

What I Read: Scaling to Multi-Terabyte Datasets

By Andrew Fairless on September 9, 2024June 29, 2024

Lessons Learned from Scaling to Multi-Terabyte Datasets v2thegreat, AliLessons Learned from Scaling to Multi-Terabyte DatasetsJune 19, 2024 “This post is meant to guide you through some of the lessons I’ve learnedContinue readingWhat I Read: Scaling to Multi-Terabyte Datasets

What I Read: string ufuncs, NumPy 2.0

By Andrew Fairless on August 27, 2024June 15, 2024

https://labs.quansight.org/blog/numpy-string-ufuncs Writing fast string ufuncs for NumPy 2.0Lysandros NikolaouPublished May 21, 2024 “After a huge amount of work from many people, NumPy 2.0.0 will soon be released, the first NumPyContinue readingWhat I Read: string ufuncs, NumPy 2.0

What I Read: PostgreSQL

By Andrew Fairless on July 23, 2024May 24, 2024

https://ottertune.com/blog/the-part-of-postgresql-we-hate-the-most The part of PostgreSQL we hate the mostBohan Zhang | Andy PavloApr 26, 2023 “But as much as we love PostgreSQL at OtterTune, certain aspects of it are notContinue readingWhat I Read: PostgreSQL

What I Read: anti-patterns, data reuse

By Andrew Fairless on July 10, 2024May 6, 2024

https://www.milesmcbain.com/posts/data-analysis-reuse Patterns and anti-patterns of data analysis reuseMiles McBainMarch 10, 2024 “When I look at that function signature I hear the screams of matrices being ground to a fine powderContinue readingWhat I Read: anti-patterns, data reuse

What I Read: predicate pushdown

By Andrew Fairless on May 29, 2024March 23, 2024

https://pola.rs/posts/predicate-pushdown-query-optimizer/ The power of predicate pushdownPolars development teamTue, 19 Mar 2024 “…we will dive into the query optimizer and explain how one of the most important optimization rules works: predicateContinue readingWhat I Read: predicate pushdown

What I Read: How fast process CSV file

By Andrew Fairless on May 8, 2024March 7, 2024

https://datapythonista.me/blog/how-fast-can-we-process-a-csv-file How fast can we process a CSV fileMarc GarciaThu 22 February 2024 “…we’ll see in this blog post how to process a CSV file as fast as possible.”

What I Read: Scaling ChatGPT, Engineering Challenges

By Andrew Fairless on April 18, 2024March 1, 2024

https://newsletter.pragmaticengineer.com/p/scaling-chatgpt Scaling ChatGPT: Five Real-World Engineering ChallengesGergely OroszFeb 20, 2024 “Just one year after its launch, ChatGPT had more than 100M weekly users. In order to meet this explosive demand,Continue readingWhat I Read: Scaling ChatGPT, Engineering Challenges

What I Read: Probabilistic Linkage, Data Deduplication

By Andrew Fairless on April 15, 2024March 1, 2024

https://www.robinlinacre.com/fellegi_sunter_accuracy/ Why Probabilistic Linkage is More Accurate than Fuzzy Matching For Data DeduplicationRobin Linacre2023-10-24 “How effectively do different approaches to record linkage exploit the information in the data to makeContinue readingWhat I Read: Probabilistic Linkage, Data Deduplication

Tag: data engineering