Synapse Spark: Add External Python Libraries
Azure Synapse Spark pools come with all the Anaconda libraries installed, but you will likely find a need for additional libraries or different versions of libraries. It is fairly simple when the required libraries exist in a repository like Conda or PyPI (Python Packaging Index). If you build your own Python WHL file (or download one) then you can add those with just a few steps. In this video I show you how to add Python libraries to your Spark pool either with session packages or workspace packages.
Check out a written tutorial here: https://dustinvannoy.com/2022/01/20/a...
More from Dustin:
Website: https://dustinvannoy.com
LinkedIn: / dustinvannoy
Twitter: / dustinvannoy
Github: https://github.com/datakickstart
CHAPTERS:
0:00 Intro
0:42 Example setup
3:00 Session Package install
4:48 environment.yml file
5:40 Workspace package install
9:23 Build Python WHL
11:32 Monitor library install
13:04 Tips
14:31 Outro
