Lakeflow Spark Declarative Pipelines - Transformation Pillar

The second Lakeflow pillar, and where the imperative streaming/MERGE code from Sections 5-8 gets replaced by declarative table definitions: core concepts (pipelines, flows, destinations), the complete syntax reference, and a full hands-on build of a real bronze -> silver -> gold pipeline – Auto Loader as a streaming table, AUTO CDC with SCD Type 2, and a gold materialized view – closing with the production judgment calls that separate a demo pipeline from one you’d trust in production.

flowchart LR
    Files[(Files)] --> BT[Streaming Table<br/>bronze]
    CDC[(CDC Feed)] --> BT
    BT --> ST[Streaming Table<br/>silver, AUTO CDC SCD2]
    ST --> MV[Materialized View<br/>gold]

Lectures

#	Lecture	Est. Read
1	What is Lakeflow SDP - Why it Replaces Manual Pipelines	8 min read
2	Core concepts - Pipelines, Flows, Destinations, and the Python API	9 min read
3	SDP API - Flow types, Destination Types, and Syntax	24 min read
4	Building the Bronze layer - Auto Loader as a Streaming Table	22 min read
5	Building the Silver layer - AUTO CDC and SCD Type 2	13 min read
6	Building the Gold layer - Materialized Views and Incremental Refresh	12 min read
7	SDP Design Decisions and Production Patterns	8 min read
8	Check Your Knowledge	Quiz

| ← Previous: Check Your Knowledge | Next: What is Lakeflow SDP - Why it Replaces Manual Pipelines → | |:—|—:|

Lakeflow Spark Declarative Pipelines - Transformation Pillar

Lectures

Table of contents