PySpark Kickstart - Read and Write Data with Apache Spark
Every Spark pipeline involves reading data from a data source or table and often ends with writing data. In this video we walk through some of the most common formats and cloud storage used for reading and writing with Spark. Includes some guidance on authenticating to ADLS, OneLake, S3, Google Cloud Storage, Azure SQL Database, and Snowflake.
Once you have watched this tutorial, go find a free dataset and try to read and write within your environment.
All thoughts and opinions are my own *
For links to the code and more information on this course, you can visit my website: https://dustinvannoy.com/2023/06/21/p...
Additional info:
Databricks read from OneLake - / integrating-microsoft-fabric-with-databricks
Integrate OneLake with Azure Databricks - https://learn.microsoft.com/en-us/fab...
Connect to Azure Storage - https://docs.databricks.com/storage/a...
Connect to AWS S3 - https://docs.databricks.com/storage/a...
Connect to Google Cloud Storage - https://docs.databricks.com/storage/g...
More from Dustin:
Website: https://dustinvannoy.com
LinkedIn: / dustinvannoy
Github: https://github.com/datakickstart
CHAPTERS
00:00 Intro
00:51 Reading CSV, JSON, Parquet
06:30 Read from Azure Storage (ADLS)
12:22 Read from AWS S3
14:22 Read from Google Cloud Storage
15:43 Read from database (Azure SQL)
19:08 Read from Snowflake
20:55 Read XML
21:57 Defining schema
24:47 Writing data (all formats)
29:18 Outro
