Azure Synapse Spark .NET (C#)
In this video, I share with you about .NET for Apache Spark. We'll walk through a quick demo on Azure Synapse Analytics, an integrated platform for analytics within Microsoft Azure cloud. This short demo is meant for those who are curious about Spark .NET or just want to get a peek at Spark in Azure Synapse. If you are new to Apache Spark, just know that it is a popular framework for data engineers that can be run in a variety of environments. It is popular because it enables distributed data processing with a relatively simple API. When developing Spark jobs, Scala or Python are the go-to languages. In the last few years several other options have emerged, but one of the newest is C#. Now, If you want to see examples in Scala or Python, you can check out one of my other videos where I walk through a similar demo.
.NET for Apache Spark is automatically available in Azure Synapse Analytics by creating a Spark notebook. You can follow along to build a Spark .NET data load that reads linked sample data, transforms data, joins to a lookup table, and saves as a Delta Lake file to your Azure Data Lake Storage Gen2 account. Please be aware that you will occur costs following this example. To keep costs minimal make the Spark pool small and keep default 15 minute auto-terminate setting.
Related Article: https://dustinvannoy.com/2021/01/27/a...
Code: https://github.com/datakickstart/syna...
Scala demo: *Coming soon
Python demo: *Coming soon
More from Dustin:
Website: dustinvannoy.com
Twitter: @dustinvannoy
Github: https://github.com/datakickstart
![Wet Hands - Minecraft (Piano Tutorial) // [Synthesia]](https://images.videosashka.com/watch/Z7ZkOTDyVeE)
