Developer Best Practices on Databricks: Git, Tests, and Automated Deployment

Опубликовано: 25 Февраль 2025
на канале: Dustin Vannoy

2,958

107

Data engineers and data scientists benefit from using best practices learned from years of software development. This video walks through 3 of the most important practices to build quality analytics solutions. It is meant to be an overview of what following these practices looks like for a Databricks developer.
This video covers:
Version control basics and demo of Git integration with Databricks workspace
Automated tests with pytest for unit testing and Databricks Workflows for integration testing
CI/CD including running tests prior to deployment with GitHub Actions

All thoughts and opinions are my own , though for this video influenced by Databricks SMEs *

Intro video that discusses development process and full list of best practices is available here: • 7 Best Practices for Development and ...

Blog post for Developer Best Practices on Databricks: https://dustinvannoy.com/2025/01/05/b...

More from Dustin:
Website: https://dustinvannoy.com
LinkedIn: / dustinvannoy
Github: https://github.com/datakickstart

CHAPTERS
0:00 Intro
0:31 Version Control (Git)
7:57 Unit Tests + Integration Tests
28:00 Automated Deploy
36:35 Outro