Derailing Is Not Failing; or, Beeminder Revenue Proportional To User Awesomeness

Akrasia
1

[adapted from a daily beemail a few months ago]

We’ve argued before that Beeminder’s incentives are not at all as perverse as they seem. If Beeminder didn’t help you succeed at your goals then you’d stop using it. We’d have to be very myopic (and certainly wouldn’t have survived 7 (!) years) to want to make people more likely to fail.

Still, it would seem too convenient to be true if maximizing Beeminder revenue and maximizing user awesomeness were always entirely identical. Well, being a fan of simultaneous having and eating of cake, I’ve got an argument for exactly that.

Here it is. As a user, your tolerance for paying Beeminder scales with the value you derive from it. And the kicks in the butt, the stings, are where all the motivation comes from. If they’re excessive then you’ll quit or make your goal too easy. Which sacrifices both revenue and awesomeness. What maximizes both revenue and awesomeness is being right below that threshold – motivating amounts of money that you’re willing to keep paying because of how much progress you’re making on your goals.

Maybe the key to seeing Beeminder’s incentives as wholly non-perverse is to remember that derailing does not equal failing. Derailing is a lot of things – a kick in the pants, paying for an immediate break because the goal has been really hard, valuable information about how realistic the goal is – none of which have anything to do with failing. The only ways to fail are to quit the goal, to set the yellow brick road to something stupidly easy, or to cheat. [1]

What do you think? I’m being serious now, and if we could convey that to users then we could unabashedly say that heck yes, Beeminder’s objective is maximize your derailments, and at the maximum amount of money you can tolerate! Put that way it sounds ridiculous but, for typical users, it’s equivalent to maximizing the amount of progress on the actual thing you’re beeminding. That’s worth paying for.

Footnotes

[1] There’s one more failure mode, thankfully rare, but it is an exception to the titular claim: Sticking your head in the sand and just letting the derailments rack up without doing any more work on the goal than you would’ve without Beeminder. When that happens it’s typically due to actual depression and we’ve got a deadman switch to bound the damage.

3 Likes
2

You can quote me on this :slight_smile: : derailing is just another data point in the journey

4 Likes
3

It’s funny I was just thinking about the original beemail version of this post in the past couple of days. Mainly, I’ve been wondering if the fact that I basically never derail on goals is a sign that I need to make some of them goals harder than they are.

3 Likes
4

Are you at capacity, or do you feel like you have extra slack where you could increase the rate and still keep up? And do you actually want to increase the rate?

1 Like
5

Yes! I’m the same way. I don’t derail very often, but I am oh so thankful that Beeminder has helped me stay on track.

For me, I could increase them and probably keep up okay. It would be tough, since I already spent half of my days just getting 2 days ahead, let alone keeping up 3 days (green). Also, in the past, I have increased these types of goals and it didn’t go well. I would get burned out just trying to keep up. I even derailed on them once or twice (gasp!).

For my goals, I do want to increase the rate, but I am only doing so slowly because I don’t want to ramp up to fast and lose all my progress or get burned out by tying myself down too much.

6

Well that’s a tough question to answer. Sometimes it’s hard to know what my capacity even is. Executive dysfunction, like nature, abhors a vacuum and it will expand to fill all the space I give it. In other words, I don’t really know if I’m at capacity until I try and then fail repeatedly.

So, yeah, I don’t know if “I’m keeping up with Beeminder so it needs to be harder” is a universalizable statement but it’s usually been true for me.

2 Likes
7

I think that it generally depends on the goal. For example I have a goal to study Kanji (Japanese alphabet) 6 minutes a day. Since it is such a small amount of time I rarely derail on it but the great thing is that I will eventually achieve the fluency that I want.

On the other hand there are goals that I rarely derail on and it is problematic because I need to put in more effort to get the results that I want. For example I use to have a goal to study programming a couple of minutes a day. I rarely derailed on the goal and that was problematic because I would not have reached the level of coding ability that I want in a reasonable time frame.

tldr: It is totally okay if you never derail on a goal as long as you can reach your primary objectives in a time frame that you want.

4 Likes
8

Agree.

9

I have a pull-ups and push-ups goal, and as soon as I reach 8 days buffer, I will increase the rate. I have not yet been able to do this :slight_smile:

4 Likes
10

Yeah, it’s hard to know until you try. I think it’s a good idea to only increase one goal, and only increase it by a little bit.

It’s also worth thinking about what you’re going to drop or give up in order to give yourself the time and energy to meet the increased rate. If you can’t identify something specific, there may not be room, or you may drop something important under the pressure.

2 Likes
11

More thoughts on this that started as thinking out loud in a daily beemail (prompted by @mary making an incisive counterpoint):

Derailing early and often until you reach your Motivation Point is great but then won’t you just continue at Peak Awesomeness without ever derailing again?

I think there are different ways it can play out. My personal favorite, and perhaps the one most compatible with this “everyone should derail as much as possible” thesis, is where you settle on an amount of money that motivates you during normal circumstances and when anything unusual happens you pay to derail and are happy about that.

It’s like a natural way to define the right amount of flexibility without having to worry about fine print (which I personally kind of dislike). The criterion is simply “are these circumstances extenuating enough that I’m ok paying this pledge?”

Of course there’s still a balancing act – you have to make sure you don’t akratically decide to derail too often.

The point is, there’s not a single Motivation Point. It depends on a million things and changes over time. But, so I claim, more derailing always correlates with Beeminder doing more motivating.

PS: And now I’m thinking about a follow-up essay about “Motivation Points vs Success Spirals” since @nick has a philosophy of beeminding that might be pretty incompatible with this whole derailing-is-not-failing thing…

3 Likes
12

It really depends on the goal. For some goals, you never want to derail, so you set the pledge really high. For others, you want the flexibility of being able to choose whether or not to derail and pay X, and you want to set X ahead of time to maximize awesomeness.

It’s not at all clear to me that the value of X that maximizes awesomeness is the same as the value of X that maximizes revenue.

Scenario 1:

Suppose 15% of the time I want to stay up late bad enough to pay $5, but not bad enough to pay $10, and 10% of the time I want to stay up late bad enough to pay $10.

At X=5, I pay $5 25% of the time, or an average of $1.25.
At X=10, I pay $10 10% of the time, or an average of $1.

Scenario 2:

Now suppose 5% of the time I want to stay up late bad enough to pay $5, but not bad enough to pay $10, and 20% of the time I want to stay up late bad enough to pay $10.

At X=5, I pay $5 25% of the time, or an average of $1.25.
At X=10, I pay $10 20% of the time, or an average of $2.

So in the first scenario, X=5 maximizes revenue, but in the second scenario, X=10 does.

Now, which maximizes awesomeness? Looking back from outside the akrasia horizon, only a small portion of your choices to pay will seem worth it. The distortion makes staying up late seem more worth it at the time than it really is taking the outside view.

So let’s say that when I would only pay $5, I regret it 60% of the time, and when I would pay $10, I regret it 40% of the time.

Then in Scenario 1 at X=5, I stay up and regret it 13% of the time and stay up and don’t regret it 12% of the time. At X=10, I stay up and regret it 4% of the time and don’t regret it 6% of the time.

In Scenario 2 at X=5, I stay up and regret it 11% of the time and stay up and don’t regret it 14% of the time. At X=10, I stay up and regret it 8% of the time and don’t regret it 12% of the time.

Now, how do we measure awesomeness? Let’s say it’s measured by the proportion of the times you don’t regret your choice minus the proportion of times you do (i.e., weighting them equally).

In Scenario 1 at X=5, awesomeness is -1%. At X=10, awesomeness is 2%.
In Scenario 2 at X=5, awesomeness is 3%. At X=10, awesomeness is 4%.

So the revenue-maximizing value of X maximizes awesomeness in Scenario 2 but doesn’t in Scenario 1.

1 Like
13

Ah, nice! This sounds right to me and I don’t think I can seriously defend a claim that maximizing awesomeness and maximizing revenue are always and necessarily the same. I think the striking thing is that they correlate at all. And as a practical matter, defining awesomeness is actually quite hard. I like your use of regret as a measure; that’s probably the right ground truth measure. But if we don’t have access to or don’t trust surveys about regret (I personally might have a really hard time answering that question for my own past behavior!) and we still need to operationalize awesomeness… it might be hard to improve on “amount paid to Beeminder” as a proxy.

1 Like
14

I really think this depends a lot on the goal. For instance, @narthur writes about his work commitments goal:

So in that case, the ideal is to never derail, and there’s no correlation between awesomeness and revenue for one derailment, but that changes if we look over the long term.

It’s worth looking at different types of derailments:

  1. For a goal where you never want to derail, but you derail due to akrasia.
  2. A goal where you set it up so that you can choose to pay $X to derail, you don’t choose to pay, but you derail anyway due to akrasia.
  3. A goal where you set it up so that you can choose to pay $X to derail, you choose to pay, and you decide in hindsight, taking the outside view, that it was worth it.
  4. A goal where you set it up so that you can choose to pay $X to derail, you choose to pay, and you decide in hindsight, taking the outside view, that it was NOT worth it - that is, you regret your choice.

With derailment type 3, there is a correlation between awesomeness of your choice and revenue. With type 4 there isn’t.

With types 1 and 2, the derailment was caused by akrasia rather than by your choice. So the relevant correlation with revenue is not single-incident awesomeness, but rather awesomeness over the entire time period preceding the derailment - like your example of the $1000 pledge paid for UVIs being worth it for the 1000 previous days.

In that kind of situation, let’s look at what happens as we increase the pledge amount X. With very low X, it won’t be motivating at all, so very little awesomeness. As we increase X, motivation increases, so the number of derailments decrease. Finally, at a certain point, making X larger will have no additional effect on motivation but will continue to increase revenue. So for example:

X      % derailment   awesomeness     avg revenue
1        50             50              0.50
10       20             80              2
100      5              95              5
1000     1              99              10
10000    1              99              100

Here awesomeness is the % of the time you do what you want, which is to always keep your commitment.

So under this model, awesomeness and revenue are correlated up to the motivation point.

2 Likes
15

Thanks for this analysis! I think we also want to model the phenomenon of making the yellow brick road too easy when the pledge is too high. I think that’s the big reason why “just risk so much you’ll never dare derail” doesn’t work.

But there are definitely cases where it’s more binary: you hit the obvious minimum or you don’t. And those are cases where awesomeness maximization means a huge pledge that you’ll definitely never pay. So max awesomeness at min revenue, sadly for Beeminder. But I guess that’s why we have premium plans! :slight_smile:

I’d actually like to collect more case studies that violate my thesis here. I mean, I’d love to believe they’re rare and that maximizing revenue is unironically the best thing we can do for our users. But loving to believe something is a pretty big red flag so y’all should push back on this!

1 Like
16

I mean, is this really a concern? I think this adds a whole new layer of complexity because it’s unclear what “too easy” means. I was assuming a fixed rate - I guess I just don’t see the rate changing as a function of the pledge. I just think the analysis is way too complicated if you add that in.

The rate is what the user puts in as what they want to accomplish - it seems a bit paternalistic to challenge that. We should define success/awesomeness in terms of the user meeting their defined goal.

Besides, even if they start with a low rate, they’ll increase the rate over time.

What makes you say it doesn’t work? I think it works once people commit - but I can see how people would be reluctant to commit.

I’m hesitant to say “you’ll definitely never pay.” No one is perfect, which is why my table maxed out at 99% success. Under that assumption, as you increase the pledge, awesomeness plateaus while revenue continues to increase.

Yeah, I can tell you really want to believe there’s a strict correspondence here. I think if you step back a bit, it’s really the other way around - the more useful Beeminder is, the more people are likely to continue to use it, upgrade, and refer others. So really, long term, the best way to maximize revenue is to maximize awesomeness.

But if you look at the choices that either the user makes or that Beeminder makes in order to maximize awesomeness for the user in the short term, those are not necessarily going to maximize revenue in the short term.

For instance, if a user pays more pledges in the short term, they’re derailing more, and Beeminder is less useful to them, and so they’re less likely to continue using it. So that’s more revenue in the short term but less in the long term.

Whereas if a user is paying fewer pledges, yes, that’s less revenue to Beeminder in the short term, but it’s more awesomeness, which means they’re more likely to continue using it, which means more long-term revenue.

1 Like
17

I’m partly disputing this part. If they leave then it means Beeminder was less useful to them. If they stay then it means the opposite.

Agreed. I’m focusing on the case that a user is paying a lot in pledge and doesn’t leave, which implies getting a lot of value out of Beeminder. If we drive them away due to derailing too much then that minimizes both awesomeness and long-term revenue and is quite compatible with my whole revenue=awesomeness thesis.

This is the key thing I’m disputing.You’re saying it’s a needless complication, paternalistic, etc, to question the slope of the yellow brick road that the user chooses. There are certainly cases where that’s true. I’m claiming that more commonly, though, when a user is never derailing it’s more likely to be because their goal is stupidly easy than because they are being super awesome. And more importantly, if a user is derailing a lot, then it must be that they’re deriving a ton of value (else they’d quit) which means doing a lot more than they’d be doing without Beeminder.

In any case, we definitely agree on the question of long-term revenue, which is the more important thing anyway. :slight_smile:

18

This is now a major motion blog post!

https://blog.beeminder.com/defail

19

this blog post is so… you, @dreev, with you and all your extremey extreminess :slight_smile:

so how about these scenarios that seem to me like counterexamples to “maximizing user awesome is equivalent to maximizing revenue”

Say I have a ukulele kicking around my house. I have a job. I have friends. I have children. I have responsibilities. I also have a vague ambition to play that ukulele around the campfire of a summer night.

If I set up a Beeminder goal to practice that uke, and cap it at $5, I’m now going to start picking up the instrument and strumming a little bit here, a little bit there. It doesn’t have to be an ambitious goal to work on my behavior. I like playing it. It’s just something that I let get bullied out of my regular day-to-day. I might derail it every once in a while, but I don’t have any interest in increasing the pledge. I don’t have any interest in increasing the time I spend on that practice either. Making the goal “harder” is actually kind of counter to my actual real-life-goals. I mean, I guess it would get me to that campfire faster, but I’m not trying to make a career of this. I’m not trying to start a band. I’m not gonna be busking in the subway. The amount of “awesomeness” that beeminder can add to my life around playing this instrument is rather binary. Doing it at all is already enough. I don’t want to make it a stressful “i have to get into juliard” kind of thing.

So in what way does “awesomeness” correlate with revenue here?

Or what about how for me, derailing is quite often failing. One of my most common derail scenarios is because I’m over capacity and I’m getting overwhelmed and things are falling through the cracks and I start derailing more and more often and what’s really happening is that I’m toeing burnout. It’s more like the slow memory leak before a core dump. So what do I do: I go archive goals and decrease slopes. So for me as a user, paying Beeminder lots of money means I dial it back.

What about here? How is revenue correlating with awesomeness here?

9 Likes
20

These are important counterexamples! For the case of ukulele practice, it’s true, I had in mind an “i have to get into julliard” kind of situation. Like all the PhD theses that have been written thanks to Beeminder (true story, for anyone just tuning in!). In those cases I think my claim holds up – if you’re not derailing then crank up the intensity!

That’s not the answer in your ukulele example, but mostly because there’s that much less awesomeness to be harvested from that example. That goal only has so much value and it makes sense that Beeminder will only be paid so much. Which means you’re right – inducing more derailing would hurt, not help.

On reflection goals like that probably are more common. Just that they also matter less than the life-changing ones. So I’m clinging to my claim that revenue=awesomeness is still mostly true, for the right definition of “mostly”. :slight_smile:

As for your next scenario, I’m tempted to say that’s just because you have your goals so well dialed in. But maybe I could use your scenario almost as a case in point. (Ok, this next part would work better if you weren’t a Beeminder founder.) You sometimes have expensive derailments and that’s during periods of non-awesomeness. Sounds like an anti-correlation! But you keep beeminding and keep pushing yourself and the only reason you’re willing to weather those expensive derailments during the non-awesome period is because of how awesome the Beeminder-augmented you is the rest of the time. So arguably the correlation is still there.

Nonetheless, it still may count as a counterexample in the sense that any inducement to more derailing would be counterproductive.

If I stop trying to be so extreme for a minute, I could scale back my claim like so:

For life-critical goals where the user has not yet honed their commitments and dialed everything in optimally, the more derailing we can encourage the better. It entails envelope-pushing and discovery that further the underlying goal.

Or maybe I’m making a God-of-the-gaps argument and as people show me more counterexamples I may have to retreat further. I’m convinced there’s something to this though. Also it might make sense to decouple this extreme revenue=awesomeness claim from “derailing is not failing”. The latter is very much true in multiple senses. Like what we talk about in http://blog.beeminder.com/seinfeld (“beeminding beats streak-minding”) and http://blog.beeminder.com/beenice (“it’s ok to treat yourself to a derailment if the cost ensures you do so sparingly”).

1 Like