Monitoring Databricks with System Tables
In this video I focus on a different side of monitoring:
What do the Databricks system tables offer me for monitoring?
How much does this overlap with the application logs and Spark metrics?
Databricks System Tables are a public preview feature that can be enabled if you have Unity Catalog on your workspace. I introduce the concept in the first 3 minutes then summarize where this is most helpful in the last 3 minutes. In between are some example queries and table explanations.
All thoughts and opinions are my own *
Blog post with more detail:
https://dustinvannoy.com/2024/02/22/d...
Options to enable system tables:
Curl - https://learn.microsoft.com/en-us/azu...
dbdemos notebook - https://www.databricks.com/resources/...
Useful blogs:
/ databricks-system-tables-an-introduction
https://www.databricks.com/blog/impro...
Useful docs:
https://learn.microsoft.com/en-us/azu...
https://learn.microsoft.com/en-us/azu...
More from Dustin:
Website: https://dustinvannoy.com
LinkedIn: / dustinvannoy
Github: https://github.com/datakickstart
CHAPTERS
0:00 Intro
1:15 Enable system tables
3:28 Billing usage query
6:55 Docs highlights
7:57 Lineage query
9:02 Audit query
11:17 Additional tables
13:20 Compare to Log4J/Metrics
14:08 Summary
