Definition
An Evolutionary Stable Strategy (ESS) is a behavioral strategy that, when adopted by most of a population, cannot be invaded by any rare alternative strategy. Introduced by John Maynard Smith and George Price in 1973, the ESS is the central concept of evolutionary game theory.
Formally: a strategy S is an ESS if, for any rare mutant strategy M, the payoff to S against S is greater than the payoff to M against S; or, if equal, the payoff to S against M is greater than the payoff to M against M. The condition ensures that a population playing S cannot be invaded by a population of M-players.
Why it matters
How it works
Consider a population in which all individuals play strategy S. A rare mutant playing strategy M arises. M can invade only if it does better against S-players than S does against itself. If not, M dies out and the population remains S-dominated.
This stability criterion is what makes ESS predictions hold up over evolutionary time. A trait observed in nature is often there not because it produces the best collective outcome, but because it is the only outcome resistant to invasion. The gap between "stable" and "optimal" is one of the field's most important insights.
The framework extends beyond pure strategies (always-cooperate, always-defect) to mixed strategies (cooperate with probability p, defect with probability 1−p) and conditional strategies (cooperate if X, defect if Y). Real biological systems often settle at mixed or conditional ESSs.