Ingestion & Streaming Decisions
Part 4: Moving & Shaping Data
Part 3 settled where data lives and how it’s shaped once it lands; Part 4, “Moving & Shaping Data,” picks up the next question — how it actually gets there. That ordering matters: choosing an ingestion cadence or a streaming architecture before the target model is settled means guessing at a destination, and this group is where the guessing stops. Its two topics work as one decision, made in sequence: first sort a source into batch, incremental, or CDC based on what it can tolerate and what access it grants; then, independent of how change is captured, decide whether that change needs to move continuously at all, or whether periodic movement already satisfies whoever’s waiting on it.
mindmap
root((Ingestion & Streaming Decisions))
("Ingestion Decisions: Batch, Incremental & CDC")
("Log-based CDC")
("Query-based CDC")
("Trigger-based CDC")
("Should This Be Streaming At All?")
("RT: real-time")
("NRT: near-real-time")
See also: From Legacy ETL to Modern ELT: Bridging Talend & Informatica-Style Tools, Batch, Near-Real-Time & Real-Time Processing: Building Robust Pipelines, Lambda Architecture: Batch + Speed Layers, and Kappa Architecture: Stream-Only Processing.
Topics
| # | Topic |
|---|---|
| 1 | Ingestion Decisions: Batch, Incremental Loading & Change Data Capture |
| 2 | Should This Be Streaming At All? RT vs NRT Trade-offs & Exactly-Once Semantics |
| ← Previous: Beyond Star Schemas: One Big Table & Wide Tables | Next: Ingestion Decisions: Batch, Incremental Loading & Change Data Capture → | |:—|—:|