Storage & I/O Foundations¶
Tier: 0 (Computational Foundations)
Concepts: 14
Durability: Timeless (rooted in physical characteristics of storage media and fundamental trade-offs in data persistence)
Fundamental Tensions¶
| Tension | Description |
|---|---|
| Durability vs. Performance | Guaranteeing data survives crashes requires forcing writes to stable storage, which is orders of magnitude slower than buffered writes. This tension exists because storage media have physical write latencies that cannot be eliminated, and buffering introduces windows of potential data loss. |
| Sequential vs. Random Access Optimization | Storage systems optimized for sequential access (logs, append-only structures) sacrifice random read efficiency, while systems optimized for random access (B-trees, indexed structures) sacrifice write efficiency. This tension arises from the physical characteristics of storage media where locality matters. |
| Latency vs. Throughput | Minimizing per-operation latency requires immediate processing, while maximizing throughput requires batching operations. You cannot simultaneously optimize both because batching inherently adds wait time. |
| Simplicity vs. I/O Efficiency | Simple I/O models (blocking, synchronous) waste CPU cycles waiting; efficient models (async, multiplexed) add significant complexity. This tension exists because hiding I/O latency requires managing concurrent operations explicitly. |
| Space vs. Time in Storage | Compact storage formats require more CPU to encode/decode; fast-access formats consume more space. Compression, indexing, and encoding decisions all navigate this fundamental trade-off. |
Concepts¶
- The I/O Cost Model
- Sequential vs. Random Access
- The Storage Hierarchy
- Buffering and Batching
- Write-Ahead Logging (WAL)
- Durability Guarantees and Fsync
- Blocking vs. Non-Blocking I/O
- I/O Multiplexing
- Read vs. Write Asymmetry
- The File Abstraction
- Crash Consistency
- Amortization of I/O Overhead
- Storage Media Characteristics
- I/O Scheduling and Prioritization
Cross-Domain Transfer Summary¶
| This Domain Concept | Transfers To |
|---|---|
| I/O Cost Model | Database design, distributed systems, web performance, any external-data-dependent system |
| Sequential vs. Random Access | Database storage engines, message queues, event sourcing, backup systems |
| Storage Hierarchy | Memory management, CDN caching, database buffers, archive storage |
| Buffering and Batching | Network protocols, database commits, graphics rendering, bulk API design |
| Write-Ahead Logging | Databases, journaling file systems, event sourcing, distributed consensus |
| Durability Guarantees | Database configuration, message queues, any crash-consistent system |
| Blocking vs. Non-Blocking I/O | Network servers, GUI programming, concurrent systems |
| I/O Multiplexing | Web servers, game servers, event systems |
| Read vs. Write Asymmetry | Database scaling, distributed systems, cache strategies |
| File Abstraction | Object storage, streams, virtual file systems |
| Crash Consistency | Transactions, consensus protocols, atomicity guarantees |
| Amortization of Overhead | Network protocols, query optimization, batch processing |
| Storage Media Characteristics | Storage engine selection, embedded systems, cloud storage |
| I/O Scheduling | Query scheduling, network QoS, multi-tenant systems |
Canonical Sources¶
| Source | Coverage |
|---|---|
| Patterson & Hennessy, "Computer Architecture: A Quantitative Approach" (multiple editions) | Storage hierarchy, I/O cost model, memory systems |
| Gray & Reuter, "Transaction Processing: Concepts and Techniques" (1992) | Durability, WAL, crash recovery, transactions |
| Stevens, "Advanced Programming in the UNIX Environment" (multiple editions) | I/O models, blocking/non-blocking, file abstractions |
| Stevens, "UNIX Network Programming" (multiple editions) | I/O multiplexing, network I/O patterns |
| Rosenblum & Ousterhout, "Log-Structured File System" (1991) | Sequential write optimization, log-structured design |
| O'Neil et al., "The Log-Structured Merge-Tree" (1996) | LSM trees, write optimization |
| Mohan et al., "ARIES: A Transaction Recovery Method" (1992) | Write-ahead logging, crash recovery |
| Hellerstein, Stonebraker & Hamilton, "Architecture of a Database System" (2007) | Database storage architecture overview |
| Jim Gray, "The Five-Minute Rule" (1987, 1997, 2007) | Storage economics, caching decisions |
| Pillai et al., "All File Systems Are Not Created Equal" (2014) | Durability guarantees, fsync behavior |
| Denning, "The Working Set Model for Program Behavior" (1968) | Working set concept, locality |
| Ritchie & Thompson, "The UNIX Time-Sharing System" (1974) | File abstraction origins |