Skip to content

Granularity of Parallelism

Work must be divided into pieces for parallel execution, but coordination has overhead. Too fine-grained: overhead dominates. Too coarse-grained: parallelism is limited. The optimal granularity depends on the cost of work versus cost of coordination.

Tension Addressed

Simplicity vs. Scalability, Isolation vs. Performance

Recognizable As

Batch sizes, chunk sizes in parallel loops, fork-join thresholds, task sizing decisions. When parallel code runs slower than sequential, granularity is often wrong. Amdahl's Law bounds speedup by sequential fraction.

Canonical Sources

Amdahl, "Validity of the Single Processor Approach" (1967); Gustafson, "Reevaluating Amdahl's Law" (1988)

Transfers To

Distributed computing (task distribution), MapReduce (split size), GPU programming (warp/block sizing)