Get the latest tech news

Goodhart’s law isn’t as useful as you might think (2023)

Goodhart's Law is useless. It tells you about a phenomenon, but it doesn't tell you how to solve it. We look at how organisations actually prevent Goodhart's Law, and illustrate this with Amazon's Weekly Business Review as an example.

During the fourth quarter (of 2000)—in which our net sales ended up increasing by 44 percent over Q4 of the previous year—there was a daily “war room” meeting where the senior Amazon leaders would analyze a three-page metrics deck and figure out what actions we’d have to take to successfully respond to the demands of what was shaping up to be a record-breaking holiday season. Repeated viewings of the same set of metrics will eventually turn into a ‘fingertip-feel’ of the business — execs will be able to say things like “this feels wrong, this dip is more than expected seasonal variation, what’s up?” — which can really only happen if you’re looking at numbers and trends every week. Instead, I argued that Donald Wheeler’s Understanding Variation gives us a more actionable formulation of Goodhart’s Law (quoting an observation from Brian Joiner, from the 80s, and drawing more broadly from the field of Statistical Process Control).

Get the Android app