Getting Started with Prefect | Task Orchestration & Data Workflows
Looking for help with your team's data strategy? → https://www.kahandatasolutions.com
Looking to improve your data engineering skillset?→ https://bit.ly/more-kds
A big challenge for many data teams is orchestrating all of the tools within their data stack.
Fortunately, there are tools designed to address this exact issue (task orchestration) and make our lives as data engineers easier.
So in today's video, I want to show you how to use a great open-source task orchestration tool called Prefect that you can use to orchestrate and monitor your entire stack from a single location.
It has a great UI, is designed with data tools in mind and only requires basic Python knowledge to get started.
By the end of this video you'll understand what Prefect is all about and how you can start using it in your own data stack.
Thank you for watching!
Timestamps:
00:00 - Intro
00:44 - What is Prefect?
02:32 - Install Prefect
04:48 - Create a Python Script
06:44 - Add a flow
07:42 - Add a task
08:45 - Add a subflow
09:49 - Intro to Deployments
12:03 - Create a Deployment
15:17 - Start an Agent
16:48 - Use Prefect Cloud
20:25 - Using Blocks
23:09 - Version Control & Storage w/ GitHub
26:07 - Automations & Task Concurrency
Title & Tags:
Getting Started with Prefect | Task Orchestration & Data Workflows
#kahandatasolutions #dataengineering #prefect
