Sharding and Partitioning¶

Horizontal scaling requires dividing data across multiple nodes; the partition key choice determines data locality, query patterns, and hotspot risk, fundamentally shaping system behavior and limitations.

Tension Addressed¶

Throughput vs. Ordering (enables parallel processing but complicates cross-partition operations)

Recognizable As¶

Partition keys; shard keys; tenant isolation; geographic partitioning; "scatter-gather" queries; cross-shard transactions (to be avoided); hotspot warnings

Canonical Sources¶

Kleppmann, "Designing Data-Intensive Applications" (2017); DeCandia et al., "Dynamo" (2007); Corbett et al., "Spanner" (2012)

Transfers To¶

Database Design (table partitioning), Multi-tenancy (tenant isolation), Search Systems (index sharding), Event Streaming (topic partitioning)

The partition key is perhaps the most consequential decision in distributed data design. It determines which operations can be local (fast) vs. distributed (slow), where hotspots will form, and what consistency guarantees are achievable within vs. across partitions.

Sub-concepts: - Range vs. Hash Partitioning: Sequential access vs. uniform distribution trade-off - Hotspot Mitigation: Key salting, compound keys, time-based rotation