Lakeflow Connect - Ingestion Pillar
The first of three Lakeflow pillars, and how data actually gets into bronze: managed SaaS connectors, query-based and CDC-based database ingestion, Auto Loader for files, and the production realities of schema drift and bad records. Closes with a direct decision guide for choosing the right tool per source.
flowchart TD
SaaS[SaaS Connector] --> B[Bronze]
QB[Query-Based Connector] --> B
CDC[CDC Connector] --> B
AL[Auto Loader] --> B
B -->|schema drift| Rescue[_rescued_data]
B -->|malformed rows| Bad[badRecordsPath]
Lectures
| # | Lecture | Est. Read |
|---|---|---|
| 1 | What is Lakeflow Connect - The Ingestion Pillar | 11 min read |
| 2 | Ingesting Data to Bronze Layer from a SaaS Application | 16 min read |
| 3 | Incremental Ingestion from Database Using Query Based Connector | 15 min read |
| 4 | Incremental CDC from Database Using Managed Connector | 10 min read |
| 5 | File-Based Ingestion with Auto Loader | 15 min read |
| 6 | Incremental Ingestion, Schema Evolution and Bad Records in Production | 12 min read |
| 7 | Choosing the Right Ingestion Tool | 4 min read |
| 8 | Check Your Knowledge | Quiz |
| ← Previous: Check Your Knowledge | Next: What is Lakeflow Connect - The Ingestion Pillar → | |:—|—:|
Table of contents
- What is Lakeflow Connect - The Ingestion Pillar
- Ingesting Data to Bronze Layer from a SaaS Application
- Incremental Ingestion from Database Using Query Based Connector
- Incremental CDC from Database Using Managed Connector
- File-Based Ingestion with Auto Loader
- Incremental Ingestion, Schema Evolution and Bad Records in Production
- Choosing the Right Ingestion Tool
- Check Your Knowledge