Having passed our SnowPro Core exam last week, we have now started on extending and enriching our data workflow. One of the tools at our disposal for this is dbt. In this blog I want to give a short introduction to the place of dbt in our workflow. Several blogs by my coworkers will give a more detailed insight in its distinct functionalities.
dbt on top of the data workflow
We have so far used Snowflake as our platform to ingest raw data, carry out transformations, and provide denormalized views to end users. Leveraging dbt provides a toolset to more easily transform the ingested data for use by the customers. As such, it is placed more or less ‘on top of’ our current data transformation step.
dbt offers several functions that aren’t (readily) available in our Snowflake data warehouse. Important examples are extensive data validation tests to verify raw data meets our expectations, iterative loops with nested SQL, and a clear track record of lineage from source data all the way to end-user views. In addition, pushing our data workflow through dbt allows for integrated git version control.
Scaling our SQL
The core principle of ELT data warehousing does not change by the introduction of transformational tools: We load data, transform it, and deliver it to the user. The introduction of the new layer allows mainly for more efficient transformations and collaborative coding. Writing and adding to function-specific documentation is also easier using dbt.
We have only scratched the surface of dbt so far, and will dive much deeper into its possibilities in preparation for our dbt Analytics Engineering exam in two months. What is already clear is that the process of transforming data in raw Snowflake was just the start: leveraging outside tools like dbt can elevate hand-written SQL pipelines to scalable, production-level solutions.
