The joined header and line items fact table makes use of stream( live.table_name) to only incrementally process new records in the upstream landing tables.
The following example shows how the orders landing table is created:Ĭreate or refresh streaming live table orders_landingĬomment "The landing orders dataset, ingested from /tmp/tahirfayyaz using auto loader."Īs select * from cloud_files("/tmp/tahirfayyaz/orders/", "json", map(Īll downstream tables can then refer to upstream tables using the stream( live.table_name) or live.table_name syntax. The landing tables make use of Auto Loader as the data source which allows you to only ingest new files landing in storage.
You can then clone the repo into your Databricks Workspace to get started in developing and deploying the DLT pipeline. If you are new to DLT you can follow the quick start tutorial to get familiar. To develop the DLT pipeline we have four Databricks notebooks structured in the following way to help you easily develop and share all of your ingestion, transformation and aggregation logic: In this guide, we will show you how to develop a Delta Live Tables pipeline to create, transform and update your Delta Lake tables and then build the matching data model in Power BI by connecting with a Databricks SQL endpoint.Īll the code for this demo is available in the Azure Databricks Essentials retail demo repo and it makes use of a TPCH dataset.