Databricks Data Engineering Project for Beginners (FREE Account) | Azure Tutorial
Get the resources (Github) 👉 https://www.lukejbyrne.com/c/azure-da...
Join the Community 👉 / @dataluke
Let's make an End-to-End Data Pipeline in Databricks.
Watch Next
• Azure End-To-End Data Engineering Pro...
• Azure Fabric End-To-End Data Engineer...
• Azure End-To-End Data Engineering Pro...
• Azure Data Factory vs Synapse vs Data...
Courses & Certificates
👇 Check them out for free 👇
Data Engineer
🌍 Understanding Data Engineering 👉 datacamp.pxf.io/JKq0ZE
🏅 Database Design 👉 datacamp.pxf.io/Xm1AEg
📊 Data Warehousing Concepts 👉 datacamp.pxf.io/WyYaPX
⚡ Introduction to dbt 👉 datacamp.pxf.io/09Y7BJ
🕹️ Introduction to Apache Airflow in Python 👉 datacamp.pxf.io/nXn3eX
📈 Introduction to NoSQL 👉 datacamp.pxf.io/6yG9RG
Cloud Engineer
🌤️ Understanding Cloud Computing 👉 datacamp.pxf.io/LKDqYY
1️⃣ Understanding Microsoft Azure 👉 datacamp.pxf.io/mOn3k1
2️⃣ AWS Concepts 👉 datacamp.pxf.io/7abrBQ
📊 Microsoft Azure Architecture and Services 👉 datacamp.pxf.io/MAQxYo
⚙️ Microsoft Azure Management and Governance 👉 datacamp.pxf.io/YRdGkj
🧩 AWS Cloud Technology and Services Concepts 👉 datacamp.pxf.io/4GdDBo
Social Media
📧 Newsletter: https://lukejbyrne.com/subscribe
🌄 Instagram: / dataluke_
⏰ TikTok: / dataluke
📚 Linkedin: / lukejbyrne
🛍️ Amazon storefront: https://www.amazon.com/shop/lukejbyrne
🏆 1:1 Coaching: https://calendly.com/lukejbyrne/30min
Some links included are affiliate links, which help me keep this channel going. Thanks for the support.
Business inquiries: [email protected]
0:00 - Introduction
3:47 - What is Databricks and its Architecture
5:33 - Medallion Architecture Explained
10:47 - Setting up Databricks in Azure
19:04 - Access Control and Security
29:04 - Connecting Databricks to Storage
36:07 - Introduction to Databricks Notebooks
44:24 - Setting up Compute Resources
45:54 - Configuring Compute Resources
48:02 - Installing Required Libraries
50:34 - Setting Up Notebooks
53:05 - Bronze Notebook: Fetching Raw Data
1:00:00 - Verifying Data in Bronze Layer
1:02:14 - Silver Notebook: Data Transformation
1:10:09 - Gold Notebook: Advanced Transformations
1:18:01 - Writing to Gold Layer
1:23:19 - Creating a Workflow in Databricks
1:27:25 - Configuring Task Dependencies
1:30:34 - Testing the Workflow
1:33:27 - Conclusion and Next Steps
