Great example of Goodhart's law - measuring the number of emails in my Archive folder

So I ran into a great example of Goodhart’s law today. Goodhart’s law is the observation that people tend to optimize their behavior for the things they measure, not for their actual goals.

It’s a problem with Beeminder and any other tool for reaching goals - it’s a general problem with setting goals and applies to every tool. It’s not something that can be overcome, except by carefully looking at your goals and your actual behavior, re-examining everything, and making changes as necessary.

So here is the example - I set up a whittle-down goal for measuring the number of emails in my inbox. I am trying to clear out my inbox and move them into storage folders. As a first step in moving the emails to storage folders, I moved them to an “archive” folder. The plan was to move them from the archive folder to specific storage folders.

So I set up another whittle-down goal for the “archive” folder. But as soon as I did that, I ran into a problem. Now every time I moved an email from inbox to archive, I was making negative progress on the archive goal - so as a result, I stopped clearing out my inbox.

When I realized what had happened, I changed the archive goal so it now measures the number of emails in my archive folder PLUS the number of emails in the inbox folder. This way, I am not disincentivized from moving an email from inbox to archive!

5 Likes

Good example and good solution!

We’ve argued on the blog that Beeminder is surprisingly robust to Goodhart’s law but I’m happy to concede that what you’ve described with your email goal is an example of it!

1 Like

@oulfis raised another good example, along with a post describing additional Beeminder goals added to try to solve the problem:

I would say Beeminder avoids the Goodhart-type problem where a manager or politician sets a bad metric and the people assessed by it can’t change it, since with Beeminder you are the one setting your own goals.

But any goal-oriented approach is going to be subject to the Goodhart-type problem where you set a goal that seems to make sense, but you end up unconsciously gaming it to some degree - which is why it’s so helpful to set up a regular review to see if your goals are actually working for you.

3 Likes