As a Databricks user, you’re likely familiar with core capabilities for big data processing and machine learning. But besides features like Spark clusters and notebooks, Databricks offers a range of powerful tools that many users don’t use. These hidden features can drastically improve your productivity, code quality, and efficiency. Databricks is more than just a “Spark notebook running platform”.
Here are five Databricks features that deserve to be used in your project.
1. Delta Live Tables (DLT)
Most Databricks users stick to using notebooks and scheduled jobs. However, Delta Live Tables can be extremely useful when building your data pipeline.
What it does: DLT allows you to declare your data transformations using simple SQL or Python, and Databricks automatically handles the orchestration, error handling, and data quality monitoring.
Why you should care: Instead of writing code for error handling, dependency management, and monitoring, you focus purely on the business logic. DLT automatically creates data lineage graphs, implements data quality checks, and handles incremental processing.
Check out the official documentation on DLT here.
2. Databricks Asset Bundles
Many teams struggle with deploying Databricks resources consistently across environments. They manually create jobs, clusters, and notebooks in each workspace, leading to different development environments which can lead to a wide range of problems.
What it does: Asset Bundles let you define your entire Databricks deployment – jobs, clusters, notebooks, libraries – as code using YAML configuration files. You can then deploy consistently across development, staging, and production environments.
Why you should care: This brings proper software engineering practices to your data workflows. You get version control for your infrastructure, reproducible deployments, and the ability to roll back changes when things go wrong.
3. Databricks SQL Serverless
Too many users default to creating compute clusters for quick analytical queries, even when they just need to run a few SQL statements. This leads to unnecessary costs and will have you waiting a long time before the cluster is provisioned.
What it does: SQL Serverless provides instant-on query execution without managing any infrastructure. It’s optimized for BI workloads and scales automatically based on the query complexity.
Why you should care: Zero wait time means your queries begin executing immediately. You pay only for what you use, and the service automatically optimizes query performance. It’s perfect for dashboards, reports, and exploratory data analysis.
4. Databricks Workflows with Multiple Tasks
Most users create jobs that run some notebooks in a certain order. However, Databricks Workflows supports multi-task jobs with conditional logic, parallel execution, and different compute requirements for each task.
What it does: You can create sophisticated workflows that combine notebooks, Python scripts, SQL queries, and even external tools. Tasks can run in parallel or sequence, with conditional branching based on certain results.
Why you should care: This eliminates the need for external orchestration tools like Airflow for some use cases. You can build complex data pipelines entirely within Databricks, with built-in monitoring, alerting, and retry logic. I’m not saying you should throw out your orchestration tool, but think of this feature the next time you’re setting up a pipeline and see how it feels.
5. Databricks Connect
This one is not a feature you should be using per sé, but it is a strong personal preference. Most users are using the default notebook interface, but Databricks Connect lets you run your favorite local IDE (VS Code, PyCharm) while executing your code on Databricks clusters. You get the best of both worlds: familiar development tools with Databricks compute power.
What it does: Connect your local development environment directly to Databricks clusters. Write code in your preferred IDE with full autocomplete, debugging, and version control, while execution happens on your Databricks cluster with access to all your data and libraries.
Why you should care: No more switching between your IDE and Databricks notebooks. You can use proper debugging tools, and version control that you know. It’s especially powerful for developing complex applications or when you need to integrate Databricks code with other systems.
Start Small
I’m not saying you should go out and implement every single feature in your Databricks project. It’s important to start small and do some low-risk experiments. Pick one feature that addresses a current issue in your workflow and implement proof of concept. Once you see the value, you can implement it further.
Which of these features will you try first? Start with the one that solves your biggest current headache, and you’ll quickly discover how much more productive and reliable your data workflows can become. I’m curious to hear if you’ve learned something from my blog. Don’t be shy and reach out to me on LinkedIn or send me an email!
