Flaky Tests¶

Tests that intermittently pass or fail without code changes indicate hidden dependencies on non-deterministic factors - timing, ordering, external state, or resource contention - and undermine confidence in the entire test suite.

Tension Addressed¶

Isolation vs. Integration

Recognizable As¶

Tests that "pass when run alone, fail in suite"; retry logic in test runners; tests disabled due to flakiness; timing-dependent assertions; tests depending on network or filesystem state

Canonical Sources¶

Luo et al., "An Empirical Analysis of Flaky Tests" (2014); Fowler, "Eradicating Non-Determinism in Tests" (2011)

Transfers To¶

Heisenbug phenomenon; reproducibility in experiments; race conditions in concurrent systems

Common sources of flakiness: - Timing/race conditions - Test order dependencies - Shared mutable state - External service dependencies - Resource exhaustion - Floating point comparison

Flaky tests are worse than missing tests because they train developers to ignore failures. A test suite with flaky tests loses its function as a reliable signal of code correctness.