Course Map

A full syllabus of The Complete Databricks Field Guide, in reading order. This page is set in a different typeface on purpose – it is a map for navigating and planning your reading, not a lesson itself. Use it to see the whole course at a glance, jump to any section directly, or track where a topic sits relative to what comes before and after it.

Part 1: Databricks Fundamentals

Core Databricks and Apache Spark concepts for a data engineer coming from a legacy warehouse or ETL background: workspace, Delta Lake, Unity Catalog, Medallion architecture, and the Lakeflow ingestion, transformation, and orchestration pillars.

• Before you start

  About the Course, Course Prerequisites, How to access Course Material and Resources

• Introduction

  Introduction to Data Engineering, Apache Spark to Data Engineering Platform, Introduction to Databricks Platform, Databricks Platform Architecture, Databricks Platform Access - Paid vs Free, Creating Databricks Free Account

• Sign up for Databricks Platform in AWS

  Overview of Databricks Premium Platform Access, Creating AWS Account, Creating AWS IAM User Account, Managing your AWS Cost, Creating AWS Databricks Service Contract, Creating AWS Databricks Serverless Workspace, Creating AWS Databricks Classic Workspace, Delete and Cleanup Databricks Workspace

• Working in Databricks Workspace

  Introduction to Databricks Workspace, Introduction to Databricks Notebooks, Notebook Magic Commands, Databricks Utilities and Widgets, How to Debug Notebooks, Workspace Files vs Git Folders, Databricks Compute Cluster, Check Your Knowledge

• Delta Lake - Deep Dive

  What is Delta Lake and why it matters, Creating and Managing Delta tables, Reading and writing Delta - batch and streaming, Time Travel - querying historical versions, OPTIMIZE, VACUUM and Data Retention, RESTORE and Rollback Strategies, MERGE INTO - the Delta Upsert Engine, DELETE, UPDATE and Idempotent writes, Schema Enforcement and Schema Evolution, Type Widening and the Variant Data Type, Table constraints - NOT NULL, CHECK, and Identity Columns, Check Your Knowledge

• Unity Catalog - Governance from day one

  Why Unity Catalog - Architecture, Implementing Unity Catalog Architecture, Catalogs, External Locations and Storage Credentials, Identity in Unity Catalog - Users, Groups, and Service Principals, Implementing Users, Groups, and Access Control, Unity Catalog Permissions Model - GRANT, Implementing Unity Catalog Permission for Catalog and Schema, Unity Catalog Permissions for Tables and Volumns, Check Your Knowledge

• Medallion Architecture - Design and Implementation

  What is Medallion Architecture, Designing the Bronze Layer, Designing the Silver Layer, Designing the Gold Layer, Architecture Review and Design Decisions, Check Your Knowledge

• Lakeflow Connect - Ingestion Pillar

  What is Lakeflow Connect - The Ingestion Pillar, Ingesting Data to Bronze Layer from a SaaS Application, Incremental Ingestion from Database Using Query Based Connector, Incremental CDC from Database Using Managed Connector, File-Based Ingestion with Auto Loader, Incremental Ingestion, Schema Evolution and Bad Records in Production, Choosing the Right Ingestion Tool, Check Your Knowledge

• Lakeflow Spark Declarative Pipelines - Transformation Pillar

  What is Lakeflow SDP - Why it Replaces Manual Pipelines, Core concepts - Pipelines, Flows, Destinations, and the Python API, SDP API - Flow types, Destination Types, and Syntax, Building the Bronze layer - Auto Loader as a Streaming Table, Building the Silver layer - AUTO CDC and SCD Type 2, Building the Gold layer - Materialized Views and Incremental Refresh, SDP Design Decisions and Production Patterns, Check Your Knowledge

• Lakeflow Jobs - The Orchestration Pillar

  What is Lakeflow Jobs - and where it fits, Your first job - pipeline task, schedule, and job-level config, Parameterizing jobs - Passing Parameters to Notebook Task, Passing Parameters to Python Script Task, Task Value, Repair Runs, Multi-task DAGs - Control Flow, Retries, and Backfill, Automating Jobs - REST API and Databricks CLI, Jobs Design Decisions and Production Patterns, Check Your Knowledge

Part 2: StepRight Capstone Project

A complete, hands-on Databricks data engineering project built end-to-end: ingestion, transformation layers, gold-layer reporting, orchestration, unit testing, data quality monitoring, and CI/CD deployment.

• StepRight - Project Foundation

  What are we building-StepRight Architecture Walkthrough, Project Structure - Planning the Initial Project Structure, Environment Setup - Repo, Catalog, Schema, and Volume, Test Data Strategy - Planning your Test Data Preparation, Seed the Project - Data Generator and Loader Notebook, Download Project Source Code

• StepRight - Ingestion Layer

  Planning and Designing Bronze Pipeline for CDC Sources, Developing Bronze Pipeline for CDC Sources, Planning and Designing Bronze Pipeline for File Sources, Developing Bronze SDP Pipeline for File Sources, Design Source Data Quality Monitoring in Bronze Layer, Developing Quarantine Pattern in Bronze Layer

• StepRight - Transformation Layers

  Design the Silver Layer Data Quality Rules and Approach, Planning and Designing Silver Pipeline for CDC Sources, Developing Silver Pipeline for AUTO CDC Sources, Planning and Designing Silver Pipeline for File Sources, Developing Silver SDP Pipeline for File Sources

• StepRight - Gold Layer Reporting and Analysis

  Revenue Computation by Day, Category, Region, Customer 360 for Marketing, Product Performance and Sales Velocity for Merchandising, Clickstream Funnel Analysis for Growth Teams, Product Delivery and Fulfilment Health for Operations

• StepRight - Orchestration and Job Scheduling

  Design the Orchestration and Job Flow, Job Pipeline Data Quality and Health Check Implementation, Job Data Quality and Health Reporting in the Job Logs, Creating the Orchestration Job

• StepRight - Unit Testing

  Design Test Strategy for the Project, Refactoring Your Code and Getting Ready for Unit Testing, Designing and Developing Unit Test Cases, Code and Run all Unit Test Case

• StepRight - Data Quality Monitoring

  Design Your Data Quality Monitoring Requirements, Setup Pipeline Event Logs for Expectations Monitoring, Prepare DQ Queries for Expectations, Quarantine and Orphans, Creating Data Quality Monitoring Dashboard, Setup a Real Time Data Quality Monitoring and Alert

• StepRight - Integration Testing, Packaging and Deployment

  Package your project using Declarative Automation Bundle, Define your Automation Bundle Resources and Setup Job, Test Your Deployment Bundle in Dev Environment, Planning and Designing Integration Testing Approach, Developing your Integration Test for the UAT, Develop and Trigger Your CI/CD Pipeline for Deployment and Integration Testing, Final Stage - Production Deployment and Smoke Testing Your Project

Part 3: Legacy Migration to Databricks

A field guide for migrating a legacy enterprise data warehouse (Oracle, Teradata, SQL Server, PL/SQL) to the Databricks Lakehouse: assessment, TCO, schema and procedure translation, reconciliation, cutover, governance migration, and FinOps.

• Why Enterprise Migrations Fail (And the Architect Who Stops It)

  The Migration Graveyard: Why 1 in 3 EDW Migrations Stall, The 80/20 Truth: What Lakebridge Does and the 20% It Never Will, The Migration Architect’s Operating Model, How This Course Works and Your Downloadable Toolkit, Check Your Knowledge

• The Autopsy: Profiling the Legacy Monolith

  The Iceberg Assessment: 20% Data, 80% Everything Else, Reading Oracle AWR Reports to Find the Real Bottlenecks, Teradata DBQL and SQL Server Query Store: Same Job, Different Dialect, Stored-Procedure Dependency Graphs and Table Heat Maps, The Workload Inventory: Your Source of Truth, Check Your Knowledge

• The 3-R Decision and the TCO That Convinces the CFO

  Rehost vs Re-platform vs Re-architect: The Decision Tree, The Migration Assessment Scorecard, Building the 3-Year TCO Calculator, Anti-Pattern: Lift-and-Shift Everything, Presenting TCO to the Board: The One Slide That Lands, Check Your Knowledge

• Lakehouse Federation: Migrate Without Migrating

  What Lakehouse Federation Actually Does, Configuring a Federated Connection to Oracle and SQL Server, The Latency and Cost Penalty: When Federation Backfires, Federate-then-Migrate: The Phased Pattern, Check Your Knowledge

• Schema Translation with Lakebridge (and Its Blind Spots)

  The Data Type Mapping Matrix, Running Lakebridge’s Analyzer and Converter on Real DDL, The Silent Precision-Loss Bug, Auditing Generated DDL for Semantic Drift, Migrating the Hard Ones: 200-Column Teradata Tables and Geospatial, Check Your Knowledge

• Physical Design for Delta: Liquid Clustering Over Indexes

  Oracle Has Muscles, Delta Has a Nervous System, Liquid Clustering vs Z-Ordering vs Partitioning, Choosing Cluster Keys from Query Patterns, Anti-Pattern: Over-Clustering Every Table, The Physical Design Decision Card, Check Your Knowledge

• The Procedure Autopsy: Decomposing PL/SQL

  Stored Procedures Are Not SQL, The Autopsy Method: Color-Coding Every Block, Only the Business Logic Migrates, Mapping a Cascading-Trigger Package, The Decomposition Worksheet, Check Your Knowledge

• Pattern Translation: Cursors, Triggers, Temp Tables, MERGE

  Cursor to Set-Based DataFrame Ops, Trigger to CDC, DBMS_SCHEDULER to Workflows, Temp Table to CTE or Lakeflow Declarative Pipeline, Oracle MERGE to Delta Lake MERGE INTO, SCD Type 2 the Lakehouse Way, Anti-Pattern Gallery and Pattern Cheat Sheet, Check Your Knowledge

• AI-Assisted Migration: Lakebridge plus LLMs With a Human Gate

  The Hybrid Model: 80% AI Speed plus 20% Human Accuracy, Writing Effective PL/SQL to PySpark Transpilation Prompts, The Window-Function Hallucination, Syntactically Valid, Semantically Wrong, The Transaction-Semantics Gap, The LLM Prompt Library and Audit Checklist, Check Your Knowledge

• The Ingestion Decision Tree

  The Data Ferry: Schedule, Capacity, Manifest, Batch vs Streaming, Push vs Pull, Schema-on-Read vs Write, JDBC Bulk vs Auto Loader: The 10TB Table Decision, Partner Tools: Fivetran, Qlik, Arcion for CDC, The Ingestion Pattern Matrix, Check Your Knowledge

• CDC and Lakeflow Declarative Pipelines With Data Contracts

  The CDC Architecture End to End, Auto Loader With Schema Inference and Evolution, Lakeflow Declarative Pipelines for SCD Type 2, Enforcing Schema at Bronze: Data Contracts, The Schema-Drift Death Spiral, Anti-Pattern: Direct-to-Silver and the Contract Template, Check Your Knowledge

• The Reconciliation Stack: Proving Semantic Parity

  Row Counts Lie: Why Semantic Parity Is the Only Truth, The 5-Layer Stack: Count, Sum, Checksum, Hash, Semantic, The $4M Bug: ROW_NUMBER Sort-Order Differences, Floating-Point Variance: SUM(DECIMAL) Oracle vs Spark, Tolerance Thresholds: Acceptable vs Real Variance, Check Your Knowledge

• Building the Reconciliation Engine

  Architecting the Reconciliation Job, The PySpark Hash-Diff Implementation, Reconciliation Dashboards: Making Parity Visible, Start at Day 0, Not End of Project, The Reconciliation Script Library, Check Your Knowledge

• The Parallel Run and Zero-Downtime Cutover

  The Parallel Universe: 30 Days of Identical Output, Running Oracle and Databricks Through a Financial Close, Cutover Patterns: Phased vs Big Bang, The Soft Cutover: Read-Only, Then Read-Write, Then Decommission, Business-User Sign-Off and the Trust Gate, Backfill and Rollback Pre-Testing, Check Your Knowledge

• Unity Catalog and the Privilege Migration

  Unity Catalog Architecture, The Security Layer Cake, The Privilege Migration Matrix, 500 Roles to 12 Tags Without a Tagging Explosion, Anti-Pattern: Replicating Oracle’s Role Hierarchy Verbatim, Check Your Knowledge

• ABAC, Masking, and Cross-Engine Governance

  ABAC With Governed Tags: Tag at Schema, Not Table, Dynamic Views and Row-Level Security, Tag-Based PII Masking, Cross-Engine Access: Delta Sharing, Iceberg, Credential Vending, The ABAC Tag Taxonomy and Masking Policy Pack, Check Your Knowledge

• The Cost Iceberg and Compute Arbitrage

  The Cost Iceberg: Visible Compute vs Hidden Everything, Compute Arbitrage: Jobs vs All-Purpose vs Serverless, Cluster Policies That Enforce Jobs Compute, The Photon Spill, Anti-Pattern: ETL on All-Purpose Because It’s Easier to Debug, Check Your Knowledge

• FinOps Engineering: System Tables and Chargeback

  The system.billing.usage Deep Dive, Attribution and Chargeback Taxonomy, Building the Chargeback Dashboard, Predictive Optimization: Proving Savings, Not Just Space, Auto-TTL: The 2026 Frontier, The System-Table Query Pack and Governance Guardrails, Check Your Knowledge

• The Migration Playbook: Sequencing the Whole Thing

  The End-to-End Migration Runbook, Sequencing: What Order, What Parallel, What Blocks What, The Risk Register: Naming Every Way It Can Fail, Stakeholder Communication and the Steering Committee, The 72-Hour Rollback Window, Check Your Knowledge

• The War Room: Capstone Simulation

  The Scenario: 50TB Oracle EDW, 500 Procs, 200 Reports, $50M Line, The Go or No-Go Decision Matrix, The 48-Hour War Room: Live, No-Go, or Rollback, The Rollback Panic, Your Capstone Deliverable and Course Wrap, Check Your Knowledge

---

| ← Previous: Home | Next: Databricks Fundamentals → | |:—|—:|