data engineering – Page 10 – Andrew Fairless, Ph.D.

What I Read: Common Errors when Debugging Airflow DAGs

By Andrew Fairless on April 14, 2021April 2, 2021

https://www.astronomer.io/blog/7-common-errors-to-check-when-debugging-airflow-dag 7 Common Errors to Check when Debugging Airflow DAGsTasks not running? DAG stuck? Logs nowhere to be found? We’ve been there. Here’s a list of common snags and someContinue readingWhat I Read: Common Errors when Debugging Airflow DAGs

What I Read: Kedro Pipelines with Airflow

By Andrew Fairless on March 30, 2021March 15, 2021

https://medium.com/quantumblack/kedro-airflow-0-4-0-orchestrating-kedro-pipelines-with-airflow-23fb1283f24d Kedro-Airflow 0.4.0 — Orchestrating Kedro Pipelines with AirflowThe Kedro team and Astronomer have released Kedro-Airflow 0.4.0 to help you develop modular, maintainable & reproducible code with orchestration superpowers!QuantumBlack, aContinue readingWhat I Read: Kedro Pipelines with Airflow

What I Read: Long Live Data Discovery

By Andrew Fairless on March 1, 2021March 1, 2021

https://towardsdatascience.com/data-catalogs-are-dead-long-live-data-discovery-a0dc8d02bd34 Data Catalogs Are Dead; Long Live Data DiscoveryWhy we need to rethink our approach to metadata management and data governanceBarr MosesDec 14 “Traditional data catalogs and governance methodologies typicallyContinue readingWhat I Read: Long Live Data Discovery

What I Watch: Future of Data Engineering

By Andrew Fairless on March 1, 2021February 27, 2021

https://youtu.be/ZZr9oE4Oa5U Future of Data EngineeringDec 16, 2019InfoQ “Chris Riccomini talks about the current state-of-the-art in data pipelines and data warehousing, and shares some of the solutions to current problems dealingContinue readingWhat I Watch: Future of Data Engineering

What I Read: Feature Store vs Data Warehouse

By Andrew Fairless on February 25, 2021February 18, 2021

https://www.logicalclocks.com/blog/feature-store-vs-data-warehouse ML Engineer Guide: Feature Store vs Data WarehouseOctober 8, 2020ML Best Practicesby Jim Dowling “The feature store is a data warehouse of features for machine learning (ML). Architecturally, itContinue readingWhat I Read: Feature Store vs Data Warehouse

What I Read: What is Data Observability?

By Andrew Fairless on February 20, 2021January 20, 2021

https://towardsdatascience.com/what-is-data-observability-40b337971e3e What is Data Observability?Hint: it’s not just data for DevOps.Barr MosesJun 25 “Instead of putting together a holistic approach to address data downtime, teams often tackle data quality andContinue readingWhat I Read: What is Data Observability?

What I Watch: Functional Data Engineering

By Andrew Fairless on February 19, 2021January 18, 2021

https://youtu.be/4Spo2QRTz1k Functional Data Engineering – A Set of Best Practices | LyftMay 24, 2018 “…we’ll discuss functional programming paradigm and explore how applying it to Data Engineering can bring aContinue readingWhat I Watch: Functional Data Engineering

What I Read: How DAGs grow

By Andrew Fairless on February 14, 2021January 18, 2021

https://greatexpectations.io/blog/deeper-wider-thicker/ How DAGs grow: Deeper, wider, and thickerNovember 16, 2020 “Over time, this turns into a steady, compounding creep in the maintenance time that you’re putting into the project…. StuffContinue readingWhat I Read: How DAGs grow

What I Read: Architectures for Modern Data Infrastructure

By Andrew Fairless on February 4, 2021January 10, 2021

https://a16z.com/2020/10/15/the-emerging-architectures-for-modern-data-infrastructure/ Emerging Architectures for Modern Data Infrastructureby Matt Bornstein, Martin Casado, and Jennifer Li “We’re now starting to see the rise of massive, complex systems built around data – whereContinue readingWhat I Read: Architectures for Modern Data Infrastructure

What I Read: data quality, ML Ops

By Andrew Fairless on January 27, 2021January 1, 2021

https://greatexpectations.io/blog/ml-ops-data-quality/ Why data quality is key to successful ML OpsIn this first post in our 2-part ML Ops series, we are going to look at ML Ops and highlight howContinue readingWhat I Read: data quality, ML Ops

Tag: data engineering