Storage & Table Formats
Part 3: Designing the Data Layer
Part 2 mapped the landscape of architecture patterns and gave you a framework for choosing among them; Part 3, “Designing the Data Layer,” is where those choices get physical — where bytes actually live, how they’re organized into something queryable, and what guarantees you can trust the instant a reader shows up. These three topics build in a straight line: object storage and file formats determine what you’re working with at the byte level, table formats layer transactional semantics on top so that byte soup behaves like an actual table, and consistency guarantees determine whether the promises those first two layers make actually hold up under concurrent, distributed access. Get this group right and every pattern from Part 2 — lakehouse, medallion, mesh — has solid ground to stand on; get it wrong and no amount of clever orchestration upstream saves you from silently wrong query results downstream.
mindmap
root((Storage & Table Formats))
("Storage Foundations: Object Storage, File Formats & Access Patterns")
("Parquet vs ORC vs Avro")
("Table Formats: Delta vs Iceberg vs Hudi")
("ACID on object storage")
("Consistency in Practice: Read-After-Write on Object Storage")
("Strong vs eventual consistency")
See also: Medallion Architecture: Bronze/Silver/Gold, ACID, BASE & the CAP Theorem: The Physics Underneath Every Data System, and Ingestion Decisions: Batch, Incremental Loading & Change Data Capture.
Topics
| # | Topic |
|---|---|
| 1 | Storage Foundations: Object Storage, File Formats & Access Patterns |
| 2 | Table Formats: Delta vs Iceberg vs Hudi |
| 3 | Consistency in Practice: Read-After-Write on Object Storage |
| ← Previous: A Mental Model for Architecture Choices: Table Formats, Cloud Providers, Hybrid Cloud & Build vs Buy | Next: Storage Foundations: Object Storage, File Formats & Access Patterns → | |:—|—:|