Parallel Load in Spark Notebook - Questions Answered
In this video I address questions from the tutorial I did on Parallel Table Ingestion with Spark Notebooks. See the chapters outline below for which questions are addressed.
Original video that questions came from:
https://dustinvannoy.com/2022/05/06/p...
Q&A Writeup:
https://dustinvannoy.com/2023/01/09/q...
Running in Scala:
https://docs.databricks.com/notebooks...
More from Dustin:
Website: https://dustinvannoy.com
LinkedIn: / dustinvannoy
Twitter: / dustinvannoy
Github: https://github.com/datakickstart
CHAPTERS:
00:00 Intro
00:40 1) How to pass multiple parameters?
07:08 2) Can I run notebooks concurrently (Databricks)?
12:20 2) Can I run notebooks concurrently (Synapse)?
14:00 3) How do we handle errors?
18:04 4) Is it using executors (workers)?
20:30 5) How does this look with Scala?
28:03 Final thoughts
