Building (Better) Data Pipelines with Apache Airflow
QCon London International Software Development Conference returns on April 8-10, 2024. Level-up on 15 major software and leadership topics including; The Tech of FinTech, What's Next in GenAI and Large Language Models (LLMs), Performance Engineering, Architecture for the Age of AI, Innovations in Data Engineering and more.
Learn the emerging trends. Explore the use cases. Implement the best practices.
Register now: https://bit.ly/3rRgsJV
--------------------------------------------------------------------------------------------------------------------------------------
In this session, Sid Anand talks about Apache Airflow, an up-and-coming platform to programmatically author, schedule, manage, and monitor workflows. Airflow’s design requires users to define DAGs (directed acyclic graphs) a.k.a. workflows in Python code, so that DAGs can be managed via the same software engineering principles and practices used to manage any other code.
With more than 7600 GitHub stars, 2400 forks, 430 contributors, 150 companies officially using it, and 4600 commits, Apache Airflow is quickly gaining traction among data science, ETL engineering, data engineering, and DevOps communities at large.
Sid Anand currently serves as PayPal's Chief Data Engineer, focusing on ways to realize the value of data.
This video was recorded at QCon.ai 2018: https://bit.ly/2piRtLl
More videos from QCon.ai 2018 on InfoQ: https://bit.ly/2rNAT8z
The InfoQ Architects' Newsletter is your monthly guide to all the topics, technologies and techniques that every professional or aspiring software architect needs to know about. Over 200,000 software architects, team leads, CTOs are subscribed to it. Sign up here: https://bit.ly/2KqYfrs
