Beeminder Forum

Derailing Is Not Failing; or, Beeminder Revenue Proportional To User Awesomeness

Well that’s a tough question to answer. Sometimes it’s hard to know what my capacity even is. Executive dysfunction, like nature, abhors a vacuum and it will expand to fill all the space I give it. In other words, I don’t really know if I’m at capacity until I try and then fail repeatedly.

So, yeah, I don’t know if “I’m keeping up with Beeminder so it needs to be harder” is a universalizable statement but it’s usually been true for me.

2 Likes

I think that it generally depends on the goal. For example I have a goal to study Kanji (Japanese alphabet) 6 minutes a day. Since it is such a small amount of time I rarely derail on it but the great thing is that I will eventually achieve the fluency that I want.

On the other hand there are goals that I rarely derail on and it is problematic because I need to put in more effort to get the results that I want. For example I use to have a goal to study programming a couple of minutes a day. I rarely derailed on the goal and that was problematic because I would not have reached the level of coding ability that I want in a reasonable time frame.

tldr: It is totally okay if you never derail on a goal as long as you can reach your primary objectives in a time frame that you want.

4 Likes

Agree.

I have a pull-ups and push-ups goal, and as soon as I reach 8 days buffer, I will increase the rate. I have not yet been able to do this :slight_smile:

4 Likes

Yeah, it’s hard to know until you try. I think it’s a good idea to only increase one goal, and only increase it by a little bit.

It’s also worth thinking about what you’re going to drop or give up in order to give yourself the time and energy to meet the increased rate. If you can’t identify something specific, there may not be room, or you may drop something important under the pressure.

2 Likes

More thoughts on this that started as thinking out loud in a daily beemail (prompted by @mary making an incisive counterpoint):

Derailing early and often until you reach your Motivation Point is great but then won’t you just continue at Peak Awesomeness without ever derailing again?

I think there are different ways it can play out. My personal favorite, and perhaps the one most compatible with this “everyone should derail as much as possible” thesis, is where you settle on an amount of money that motivates you during normal circumstances and when anything unusual happens you pay to derail and are happy about that.

It’s like a natural way to define the right amount of flexibility without having to worry about fine print (which I personally kind of dislike). The criterion is simply “are these circumstances extenuating enough that I’m ok paying this pledge?”

Of course there’s still a balancing act – you have to make sure you don’t akratically decide to derail too often.

The point is, there’s not a single Motivation Point. It depends on a million things and changes over time. But, so I claim, more derailing always correlates with Beeminder doing more motivating.

PS: And now I’m thinking about a follow-up essay about “Motivation Points vs Success Spirals” since @nick has a philosophy of beeminding that might be pretty incompatible with this whole derailing-is-not-failing thing…

3 Likes

It really depends on the goal. For some goals, you never want to derail, so you set the pledge really high. For others, you want the flexibility of being able to choose whether or not to derail and pay X, and you want to set X ahead of time to maximize awesomeness.

It’s not at all clear to me that the value of X that maximizes awesomeness is the same as the value of X that maximizes revenue.

Scenario 1:

Suppose 15% of the time I want to stay up late bad enough to pay $5, but not bad enough to pay $10, and 10% of the time I want to stay up late bad enough to pay $10.

At X=5, I pay $5 25% of the time, or an average of $1.25.
At X=10, I pay $10 10% of the time, or an average of $1.

Scenario 2:

Now suppose 5% of the time I want to stay up late bad enough to pay $5, but not bad enough to pay $10, and 20% of the time I want to stay up late bad enough to pay $10.

At X=5, I pay $5 25% of the time, or an average of $1.25.
At X=10, I pay $10 20% of the time, or an average of $2.

So in the first scenario, X=5 maximizes revenue, but in the second scenario, X=10 does.

Now, which maximizes awesomeness? Looking back from outside the akrasia horizon, only a small portion of your choices to pay will seem worth it. The distortion makes staying up late seem more worth it at the time than it really is taking the outside view.

So let’s say that when I would only pay $5, I regret it 60% of the time, and when I would pay $10, I regret it 40% of the time.

Then in Scenario 1 at X=5, I stay up and regret it 13% of the time and stay up and don’t regret it 12% of the time. At X=10, I stay up and regret it 4% of the time and don’t regret it 6% of the time.

In Scenario 2 at X=5, I stay up and regret it 11% of the time and stay up and don’t regret it 14% of the time. At X=10, I stay up and regret it 8% of the time and don’t regret it 12% of the time.

Now, how do we measure awesomeness? Let’s say it’s measured by the proportion of the times you don’t regret your choice minus the proportion of times you do (i.e., weighting them equally).

In Scenario 1 at X=5, awesomeness is -1%. At X=10, awesomeness is 2%.
In Scenario 2 at X=5, awesomeness is 3%. At X=10, awesomeness is 4%.

So the revenue-maximizing value of X maximizes awesomeness in Scenario 2 but doesn’t in Scenario 1.

2 Likes

Ah, nice! This sounds right to me and I don’t think I can seriously defend a claim that maximizing awesomeness and maximizing revenue are always and necessarily the same. I think the striking thing is that they correlate at all. And as a practical matter, defining awesomeness is actually quite hard. I like your use of regret as a measure; that’s probably the right ground truth measure. But if we don’t have access to or don’t trust surveys about regret (I personally might have a really hard time answering that question for my own past behavior!) and we still need to operationalize awesomeness… it might be hard to improve on “amount paid to Beeminder” as a proxy.

2 Likes

I really think this depends a lot on the goal. For instance, @narthur writes about his work commitments goal:

So in that case, the ideal is to never derail, and there’s no correlation between awesomeness and revenue for one derailment, but that changes if we look over the long term.

It’s worth looking at different types of derailments:

  1. For a goal where you never want to derail, but you derail due to akrasia.
  2. A goal where you set it up so that you can choose to pay $X to derail, you don’t choose to pay, but you derail anyway due to akrasia.
  3. A goal where you set it up so that you can choose to pay $X to derail, you choose to pay, and you decide in hindsight, taking the outside view, that it was worth it.
  4. A goal where you set it up so that you can choose to pay $X to derail, you choose to pay, and you decide in hindsight, taking the outside view, that it was NOT worth it - that is, you regret your choice.

With derailment type 3, there is a correlation between awesomeness of your choice and revenue. With type 4 there isn’t.

With types 1 and 2, the derailment was caused by akrasia rather than by your choice. So the relevant correlation with revenue is not single-incident awesomeness, but rather awesomeness over the entire time period preceding the derailment - like your example of the $1000 pledge paid for UVIs being worth it for the 1000 previous days.

In that kind of situation, let’s look at what happens as we increase the pledge amount X. With very low X, it won’t be motivating at all, so very little awesomeness. As we increase X, motivation increases, so the number of derailments decrease. Finally, at a certain point, making X larger will have no additional effect on motivation but will continue to increase revenue. So for example:

X      % derailment   awesomeness     avg revenue
1        50             50              0.50
10       20             80              2
100      5              95              5
1000     1              99              10
10000    1              99              100

Here awesomeness is the % of the time you do what you want, which is to always keep your commitment.

So under this model, awesomeness and revenue are correlated up to the motivation point.

3 Likes

Thanks for this analysis! I think we also want to model the phenomenon of making the yellow brick road too easy when the pledge is too high. I think that’s the big reason why “just risk so much you’ll never dare derail” doesn’t work.

But there are definitely cases where it’s more binary: you hit the obvious minimum or you don’t. And those are cases where awesomeness maximization means a huge pledge that you’ll definitely never pay. So max awesomeness at min revenue, sadly for Beeminder. But I guess that’s why we have premium plans! :slight_smile:

I’d actually like to collect more case studies that violate my thesis here. I mean, I’d love to believe they’re rare and that maximizing revenue is unironically the best thing we can do for our users. But loving to believe something is a pretty big red flag so y’all should push back on this!

2 Likes

I mean, is this really a concern? I think this adds a whole new layer of complexity because it’s unclear what “too easy” means. I was assuming a fixed rate - I guess I just don’t see the rate changing as a function of the pledge. I just think the analysis is way too complicated if you add that in.

The rate is what the user puts in as what they want to accomplish - it seems a bit paternalistic to challenge that. We should define success/awesomeness in terms of the user meeting their defined goal.

Besides, even if they start with a low rate, they’ll increase the rate over time.

What makes you say it doesn’t work? I think it works once people commit - but I can see how people would be reluctant to commit.

I’m hesitant to say “you’ll definitely never pay.” No one is perfect, which is why my table maxed out at 99% success. Under that assumption, as you increase the pledge, awesomeness plateaus while revenue continues to increase.

Yeah, I can tell you really want to believe there’s a strict correspondence here. I think if you step back a bit, it’s really the other way around - the more useful Beeminder is, the more people are likely to continue to use it, upgrade, and refer others. So really, long term, the best way to maximize revenue is to maximize awesomeness.

But if you look at the choices that either the user makes or that Beeminder makes in order to maximize awesomeness for the user in the short term, those are not necessarily going to maximize revenue in the short term.

For instance, if a user pays more pledges in the short term, they’re derailing more, and Beeminder is less useful to them, and so they’re less likely to continue using it. So that’s more revenue in the short term but less in the long term.

Whereas if a user is paying fewer pledges, yes, that’s less revenue to Beeminder in the short term, but it’s more awesomeness, which means they’re more likely to continue using it, which means more long-term revenue.

1 Like

I’m partly disputing this part. If they leave then it means Beeminder was less useful to them. If they stay then it means the opposite.

Agreed. I’m focusing on the case that a user is paying a lot in pledge and doesn’t leave, which implies getting a lot of value out of Beeminder. If we drive them away due to derailing too much then that minimizes both awesomeness and long-term revenue and is quite compatible with my whole revenue=awesomeness thesis.

This is the key thing I’m disputing.You’re saying it’s a needless complication, paternalistic, etc, to question the slope of the yellow brick road that the user chooses. There are certainly cases where that’s true. I’m claiming that more commonly, though, when a user is never derailing it’s more likely to be because their goal is stupidly easy than because they are being super awesome. And more importantly, if a user is derailing a lot, then it must be that they’re deriving a ton of value (else they’d quit) which means doing a lot more than they’d be doing without Beeminder.

In any case, we definitely agree on the question of long-term revenue, which is the more important thing anyway. :slight_smile:

This is now a major motion blog post!

https://blog.beeminder.com/defail

this blog post is so… you, @dreev, with you and all your extremey extreminess :slight_smile:

so how about these scenarios that seem to me like counterexamples to “maximizing user awesome is equivalent to maximizing revenue”

Say I have a ukulele kicking around my house. I have a job. I have friends. I have children. I have responsibilities. I also have a vague ambition to play that ukulele around the campfire of a summer night.

If I set up a Beeminder goal to practice that uke, and cap it at $5, I’m now going to start picking up the instrument and strumming a little bit here, a little bit there. It doesn’t have to be an ambitious goal to work on my behavior. I like playing it. It’s just something that I let get bullied out of my regular day-to-day. I might derail it every once in a while, but I don’t have any interest in increasing the pledge. I don’t have any interest in increasing the time I spend on that practice either. Making the goal “harder” is actually kind of counter to my actual real-life-goals. I mean, I guess it would get me to that campfire faster, but I’m not trying to make a career of this. I’m not trying to start a band. I’m not gonna be busking in the subway. The amount of “awesomeness” that beeminder can add to my life around playing this instrument is rather binary. Doing it at all is already enough. I don’t want to make it a stressful “i have to get into juliard” kind of thing.

So in what way does “awesomeness” correlate with revenue here?

Or what about how for me, derailing is quite often failing. One of my most common derail scenarios is because I’m over capacity and I’m getting overwhelmed and things are falling through the cracks and I start derailing more and more often and what’s really happening is that I’m toeing burnout. It’s more like the slow memory leak before a core dump. So what do I do: I go archive goals and decrease slopes. So for me as a user, paying Beeminder lots of money means I dial it back.

What about here? How is revenue correlating with awesomeness here?

8 Likes

These are important counterexamples! For the case of ukulele practice, it’s true, I had in mind an “i have to get into julliard” kind of situation. Like all the PhD theses that have been written thanks to Beeminder (true story, for anyone just tuning in!). In those cases I think my claim holds up – if you’re not derailing then crank up the intensity!

That’s not the answer in your ukulele example, but mostly because there’s that much less awesomeness to be harvested from that example. That goal only has so much value and it makes sense that Beeminder will only be paid so much. Which means you’re right – inducing more derailing would hurt, not help.

On reflection goals like that probably are more common. Just that they also matter less than the life-changing ones. So I’m clinging to my claim that revenue=awesomeness is still mostly true, for the right definition of “mostly”. :slight_smile:

As for your next scenario, I’m tempted to say that’s just because you have your goals so well dialed in. But maybe I could use your scenario almost as a case in point. (Ok, this next part would work better if you weren’t a Beeminder founder.) You sometimes have expensive derailments and that’s during periods of non-awesomeness. Sounds like an anti-correlation! But you keep beeminding and keep pushing yourself and the only reason you’re willing to weather those expensive derailments during the non-awesome period is because of how awesome the Beeminder-augmented you is the rest of the time. So arguably the correlation is still there.

Nonetheless, it still may count as a counterexample in the sense that any inducement to more derailing would be counterproductive.

If I stop trying to be so extreme for a minute, I could scale back my claim like so:

For life-critical goals where the user has not yet honed their commitments and dialed everything in optimally, the more derailing we can encourage the better. It entails envelope-pushing and discovery that further the underlying goal.

Or maybe I’m making a God-of-the-gaps argument and as people show me more counterexamples I may have to retreat further. I’m convinced there’s something to this though. Also it might make sense to decouple this extreme revenue=awesomeness claim from “derailing is not failing”. The latter is very much true in multiple senses. Like what we talk about in http://blog.beeminder.com/seinfeld (“beeminding beats streak-minding”) and http://blog.beeminder.com/beenice (“it’s ok to treat yourself to a derailment if the cost ensures you do so sparingly”).

Maybe the key to seeing Beeminder’s incentives as wholly non-perverse is to remember that derailing does not equal failing. Derailing is a lot of things — a kick in the pants, paying for an immediate break because the goal has been really hard, valuable information about how realistic the goal is — none of which have anything to do with failing. The only ways to fail are to quit the goal, to set the yellow brick road to something stupidly easy, or to cheat.

That’s also the key of the article: This presupposes a certain definition of “failing”. And then goes on to say that derailing is no such thing. In my opinion what failing is, is a very subjective question that can’t really be dictated by Beeminder, because it’s an opinion (and emotions-) based matter and might change not only from individual to individual but also from goal to goal, day to day, etc.

I know Beeminder is a company and needs or at least wants to have a set of applicable principles (a (predictive) theory of beeminder) to gauge stuff like revenue and user engagement, but I think this is also a prime case of removing unknowables by presupposing certain definitions and then working under the assumption that the unknowables now have become knowable. I would even suggest that many users couldn’t even articulate why something feels like a failure, which makes following @dreev’s definition even harder.

To say it in an image: The article claims to be a map of a territory but it’s actually a blueprint for a terraforming project, since it doesn’t describe the landscape which is unexplored. Instead it draws up a class of use cases and tries to rework the discursive landscape in which it acts (ergo: convince other people that what they see is indeed what @dreev describes if we only put a fresh coat of paint on it. And remove this detail. And add this extra thing over here. And so on.). :wink:

But also that’s basically how you get to a predictive theory of anything: You control the discourse by standardization (of definitions) and so on… So I don’t even know if it’s a bad thing in that sense, but we also loose complexity (and in this case a lot of it) by clearing things up in that way…

6 Likes

This is confusing short-term and long-term. If they’re derailing a lot, that’s high revenue short term, which might or might not mean high awesomeness. They might be derailing a lot but not deriving much value, in which case, yes, they would leave after a while, but in the meantime you have (short-term) high revenue and low awesomeness.

There are different kinds of awesomeness we can talk about.
First is “gross awesomeness.” Using Bee’s ukulele example, gross awesomeness is how much you practice ukulele.

But that doesn’t take into account the downsides, of which there are two:

First, the opportunity cost of all the other things you might prefer doing. As Bee points out, doing it too much will actually decrease awesomeness because it adds stress and takes away time from other things in your life.

And second, the pledge money you have to pay. If you end up practicing a lot, but paying so much in pledges that it’s not worth it to you, beeminder may get a lot of revenue, but the net awesomeness to the user is little or negative.

The user wants to choose a rate and pledge that maximizes net awesomeness, given their level of akrasia.

3 Likes

@matti: Dang, there’s a lot of insight packed in to that. Thank you! I’m chewing particularly hard on this “not a map of the territory but a terraforming project” concept handle you just gave me.

I think I’ve added a bunch of confusion talking about long-term vs short-term revenue, the joke about running away to Mexico, etc, but in the blog post and all my arguments here when I talk about “Beeminder revenue” I always mean overall, long-term revenue.

The case of high short-term revenue and low awesomeness is simply a case of low revenue in the important sense.

Now we’re just quibbling about how to divvy the consumer surplus. :slight_smile: That question is always secondary to the question of social/economic efficiency.

1 Like

Well if we’re only talking about long-term:

In my view the optimal amount of derailing for the user is none or low, whereas the amount of derailing that maximizes revenue for beeminder is somewhat higher.

Too much derailing means your pledge is too low or your rate is too high. But more derailing means more revenue for beeminder as long as it’s sustainable.

  • There’s an exception: If the pledge is very low, more frequent derailing might actually mean less revenue than less frequent derailing at a higher pledge, but this isn’t guaranteed - it depends on how the user’s motivation increases with money. If there’s a sharp cutoff at the motivation point (say $100), user awesomeness is maximized when the pledge is just above the motivation point ($101), whereas revenue for beeminder is maximized when the pledge is either insanely high ($10,000) or sustainably below the motivation point ($45). $75 might not be sustainable, and $25 might be too low revenue.)

In my view, as derailing increases, once you’re past the point when you’re choosing to derail once in a while (beenice), awesomeness for the user starts decreasing, yet (long-term) beeminder revenue continues to increase - up until we get to derailing levels so high that they are not sustainable because the user quits out of frustration.

  • No/little derailing - could mean either high awesomeness because you’re at your motivation point, or medium awesomeness because your rate is below optimal.

    — Corresponds to medium income for beeminder long-term.

  • Medium derailing - could mean derailing is by choice, but then your rate is probably too high, so low awesomeness. could mean your pledge is below optimal, so low awesomeness. Users will probably not get too frustrated, though, so this is probably sustainable. Could mean it’s not the right goal for you.

    — Corresponds to high income for beeminder long-term, if sustainable.

  • Lots of derailing - probably means beeminder is not working for you. Pledge may be below optimal, rate may be too high, or not the right goal for you. Likely to not be sustainable and lead to frustration.

    — Corresponds to low income for beeminder long-term because it isn’t sustainable.

Each user has a level and type of akrasia corresponding to a function of two variables - frequency of keeping your pledge is a function of the pledge and rate.

  • f = f(p,r)

Beeminder revenue is pledge times frequency of not keeping your pledge.

  • BR = p(1-f) = p-pf

For each user and goal there is an ideal rate i. The rate the user actually performs will be the rate they set times the proportion of times they don’t derail, or rf. The user wants this to be as close to i as possible, that is, to minimize |rf - i|. The user also wants to minimize payment to beeminder, that is, minimize p-pf.

So we can model awesomeness to the user as a(p-pf) + b(|rf - i|). The maximum awesomeness is 0. a is a negative constant corresponding to how much it sucks to pay each dollar to beeminder, and b is a negative constant corresponding to how much it sucks to do the task more or less than the ideal rate.

@dreev’s theory that revenue is proportional to awesomeness can be expressed as c(p-pf) = a(p-pf) + b(|rf-i|).

This can be rewritten as:

(1)         n(p-pf) = |rf-i|

where n is the new constant (c-a)/b.

(1) essentially is a claim about akrasia - that akrasia works in a way such that users’ akrasia functions generally satisfy, or come close to satisfying, (1).

2 Likes

To make the math a little easier we can instead assume the user wants to minimize (rf - i)^2. We then have:

(2)     n(p-pf) = (rf-i)^2

We can rewrite this as np-npf = r^2 f^2 - 2rfi + i^2

We can write that as (r^2) f^2 + (np-2ri) f + (i^2 - np) = 0

Which by the quadratic formula means f(p,r) satisfies

(3) f(p,r) = (2ri-np) / (2 r^2)  + or - sqrt( (np-2ri)^2 - 4 r^2 (i^2 - np) ) / (2 r^2)

What would be really nice here is a Mathematica 3D graph of f(p,r) with sliders to change n and i so we can see what this surface looks like!

1 Like