Databricks Monitoring with Log Analytics - Updated for DBR 11.3+
In this video I show the latest way to setup and use Log Analytics for storing and querying you Databricks logs. My prior video covered the steps for earlier Databricks Runtime Versions (prior to 11.0). This video covers using the updated code for Databricks Runtime 11.3, 12.2, or 13.3.
There are various options for monitoring Databricks, but since Log Analytics provides a way to easily query logs and setup alerts in Azure you may choose to send your Databricks logs there as well.
All thoughts and opinions are my own *
Blog post with more detail:
https://dustinvannoy.com/2024/01/07/m...
References:
spark-monitoring repo: https://github.com/mspnp/spark-monito...
Data & AI Summit Presentation: • Simplifying Lakehouse Observability: ...
More from Dustin:
Website: https://dustinvannoy.com
LinkedIn: / dustinvannoy
Github: https://github.com/datakickstart
CHAPTERS
00:00 Intro
01:46 Requirements and overview
05:30 Demo - Build + Upload
10:53 Demo - Deploy
13:22 Demo - Query
15:42 Troubleshooting
16:45 Outro
