A new meta goal concept

[scarabaea]

I have had a meta goal for keeping ahead of the road across my goals for a while. This way I can make sure that I still do the average on a list of goals but I get to choose what to do more of on a specific day. Plus helps to avoid the stress of having every single one in the red every day and burning the buffer due to acrasia rather than saving it for emergencies. This post is not to argue why something like this is useful (I know there are various modes of using Beeminder) but to suggest a new concept of how to implement that meta goal.

For years, I have been doing it by summing the “days until derailment,” first manually, then, since recently, using API and Python. Here I described my approach in my personal journaling thread.

What I always found annoying about using the days-until-derailment as a proxy is how such a meta goal behaves around flat spots / lower-rate spots / post-derailment respites. The thing is, if I am really interested in how far ahead of the road I am, it is “how far above the road” I am, because progress in Beeminder graphs is moving up. And when we have monotonously increasing (do-more or odometer) or monotonously decreasing (whittle-down) goals, no upcoming changes in the road, simple geometry suggests that “how far above the road the latest datapoint is” is absolutely synonymous to “how soon a horizontal line from the latest datapoint crashes into the road unless more progress is done”. Works perfectly.

But as soon as there are any changes in the road planned, these things stop to be the same, geometrically speaking. Consequently:

1. a derailment immediately ups the meta goal with the number equal to the number of days of the post-derailment respite. So, a derailment on one goal automatically decreases the positive effect of the meta goal on every other goal in the list on the day after derailment.
2. introduced flat spots or lower-rate spots actually start affecting the meta goal as many days ahead of the planned break as I have “days until derailment on that goal”. Which is not how a flat spot should work. If I add one for travel, I want a lowered load exactly on the days when I travel, not a week before. But this is what I get currently. And then the normal load instead of the lowered load on the last days of the break, as far as the goal’s impact on the meta goal is concerned.
3. because of #2, it is impossible to time a break on the meta goal for such days away like when travelling, so I necessarily derail every time after such days.

While using days-until-derailment had the upside of using an easily available, already visibly displayed value when I was still counting the sum manually, now that I am having Python calculate it for me anyways, it’s not that important anymore.

So, I came up with an implementation that actually takes into account how far ahead of plan = above the road I am. Thus, the flat spots work when they are expected to, not several days earlier. The tricky question was what to use as the baseline value when converting the vertical distance from the road into “days ahead of plan”. I decided that it will be the highest rate for all segments between today and the goal end date. This is based on the assumption (that might be not true for other use cases) that there is some rate which is normally in use for a goal, which is only lowered to 0 or to a decreased number in specific situations, but otherwise it is the rate that the user strives to. I.e. the lower-rate spots are treated as abnormalities that will be eventually over. (Well, if someone is on a higher-rate spot at some point, then when it ends (and the rate for the remaining lifetime of the goal is lower), nothing awful will happen either, there will be just an extra boost to the meta goal on the next day. Like the #1 consequence above in the previous-concept meta goal, but by definition much rarer than upon every derailment.)

Anyways, if anybody would like to also use my idea, here’s the link to my repository where I posted an anonymized copy of my code (co-authored by ChatGPT). Considerations about usage copied from “readme” part there:

This code helps calculate the current value for a meta goal based on how far above the road are latest datapoints across a list of goals.

Before using, mark the list of the goals which you need to take into account with a unique tag. (It only makes sense to take into account some goals and not others. For example, it doesn’t work for do-less goals. Works best for goals where the direction of data accumulation is monotonous. E.g. odometer or do-more goals where the total can only become higher and whittle-down goals where the total can only become lower. Should work fine with odometer resets.)

Set up a goal that will accept the meta values as an odometer goal. I use the rate of 0.1, just for it to be non-zero, but the rate is not important here, since the pressure will be not from this goal’s rate but from the rates of all the goals that are taken into account.

The datapoint that is posted takes into account 10 goals with closest-to-the-road latest datapoints. Think of it as a score on the scale of 0 to 100, where 0 is “at least 10 goals are in the red today” and 100 as “all goals are at least 10 days ahead of the plan”. In my experience, it make little sense to actually strive to achieve the 90+ range, as then the freedom to work more one this or that goal disappears. Instead, it is the journey that is the destination

Add your INFO in the second cell with code.

After the last cell, you can see a recap confirming that the datapoint has been posted and a list of goals that you can work on that contribute to the meta goal.

[scarabaea]

Here’s an example:

The goals for which the penultimate column is the same as the number of days in the last column are those where no road changes are expected in the nearest future.
Those where it is lower than the number of days in the last column - those are where an upcoming lower-rate spot is already soon enough to meddle with the adequacy of the days-until-derailment as a proxy for how far ahead I am. Especially see the last rows, for the goals where lengthy flat spots are upcoming.

[poisson]

Interesting idea!

Have you considered how one would handle do-less goals? I guess you chose not to because you didn’t want to anyway for any of your particular goals? It isn’t obvious to me why it couldn’t be made to work for them, however. Although the general screwed-upness of breaks for do-less goals probably carries over into some degree of screwed-upness for this too…

[scarabaea]

Yes, I think it should be possible to include the do-less goals. It’s just that I don’t have any, so I didn’t need for my usage. So far, the calculation depends on the slope going the same direction where the datapoints are compared to the road (up and above for do-more and odo; down and below for whittle-down). A separate calculation just needs to be incorporated for the case of do-less, where the road goes up but the points are below it. Thanks for the suggestion, I might do that next.

[scarabaea]

I uploaded a new version to Github. Some changes:

1. Settings for username etc are now in a separate file
2. It’s possible to include do-less goals now (@poisson). Though my philosophy for this type of meta goal is still not to include any goals where the datapoints can move closer to the road (including odometer or whittle-down where the number can fluctuate both ways)
3. In the dataframe that is displayed in the end of all the calculations, it is possible to show all goals, not only the ones that contribute to the meta goal. So this can be used as a working dashboard to see how much still needs to be done.
4. Color highlighting:
   
   

[poisson]

Looks great!

[lanthala]

That’s really cool! I’m not sure it quite crosses the threshold to be worth whatever fresh hell I will be placed in if I try to get it working, but I do like the idea.

I particularly like the idea of accumulating buffer granting you the flexibility to work on things when you want to; right now that’s definitely my biggest struggle with beeminder. I hate days when I have 5 nontrivial goals due that evening, derailing any other plans I had made. But the sorting options don’t make it easy for me to see when that’s coming up.

I was imagining the solution would be a “custom coloring” for different goals (so my anki goal sorts as “red” when it has one day of buffer, while my water plants goal sorts as “red” when it has 0 days of buffer) but metaminding specific goals (where it makes sense to work ahead, unlike watering the plants) could also work.

[alephnull]

Big fan of this idea! I’ve had an urgency load goal for a while now. It’s great overall as an impetus to work on goals that aren’t due today, but the “vacation problem” means I’ll treat some goals as completely in the clear when I really should be planning now for the post-vacation slope. So I’m excited about this as a potentially better metric!

I downloaded your code and started tracking “days ahead” in addition to urgency load, to see if I want to switch.

Right now my biggest issue is that “max-future-rate” doesn’t work well with how I use Beeminder: I sometimes like to schedule rate changes far in the future (e.g. I’ll run more in the spring than in the winter, so I’ll schedule a rate increase a few months in advance to prevent me from conveniently forgetting later on).

My thoughts as I tried to address this:

• Changing this to “max rate between now and derailment” would be nice, but it has the unfortunate property that doing more work on a goal could decrease your days ahead (e.g. if you have a large rate increase coming up)
• Ok, why not compute “days ahead” independently for each road segment and add those up? … oh, I’ve just reinvented “days until derailment.” (But worse, since now “days ahead” is undefined / infinite for a flat segment.)
• Can I make flat segments a special case? Maybe, but it’s a bit arbitrary, and you’d prefer for very-shallow-slope segments to work similarly to flat segments. (Although “urgency load but ignoring flat segments” might be an improvement over regular urgency load?)

I eventually came around to thinking “max future rate” is less bad than my alternatives, with “max rate in next D days” as perhaps a slight improvement.

The downside is that D is arbitrary and it makes the metric kind of wrong around a rate change (but there’s no way to avoid being wrong in some way or another). The upside is that it can prepare you for an upcoming rate increase in advance, which might be nice.

Just for fun, here’s the formula written out. Choose values

• k = number of goals to include
• M = max value per goal
• D = number of days to look forward for max slope

Then \text{Days-Ahead}(k, M, D) = \displaystyle \sum_{g \in \{\text{next }k \text{ goals due}\}} \min\left(M,\, \frac{\text{buffer}(g)}{\text{max-rate}_D(g)}\right).

The scarabaea version of this would be \text{Days-Ahead}(10, 10, \infty).

I’ve spent too long on this for today, but if I get around to implementing this in the code I’ll post my version here.

[scarabaea]

I am happy that you found my concept inspiring and are adapting it to your beeminding practices.

What I would do in such a case… I would just manually add buffer (via introducing a flat spot, or with new goals I now start with an initial dummy datapoint = 8 days of the rate, instead of the horizontal buffer offer at goal creation) and then just look at what the script returns, taking the harder rate into consideration. The “days above” and the “days until derailment” will be very different, but if it’s something that I don’t rely on being in the red/orange on daily basis, it works okay. So, I would just interpret the “spring” rate as the normal rate and the “winter” rate (in your example) as a prolonged shallower-rate spot.

[alephnull]

Thanks for the suggestion!

I’d like to be able to use the per-goal “days ahead” number to help me prioritize. (My main Beeminder interface is a modified version of @alys’s command-line UI, and I’m thinking of adding an option to sort by “days ahead” if I can get the metric to feel right.)

The place where I feel this most is with working hours (which I track with weekends-off enabled). I like to stay about 3 days ahead, ideally. If it’s Wednesday, this will show up as having a 5-day safety buffer, so unless I consciously remind myself that this includes the weekend, I feel like I’m further ahead than I actually am and risk dipping into my buffer. If I’m measuring days ahead, I can try to keep this at 3 all the time.

Taking a far-future rate into account messes this up: I have to consciously remind myself that it’s using a different rate, so it’s less useful for prioritizing what to do today.

I think the rolling-window strategy will fix this for my purposes (but I’ll have to see how it looks when I get it going). My hope is that by setting D somewhere around 10-20ish, it’ll be long enough to capture whatever “real rate” you’re using at the time but short enough that the individual goal values feel relevant. (Something to watch out for is a break of more than D days, which could break it.)

[alltom]

@alephnull

• Ok, why not compute “days ahead” independently for each road segment and add those up? … oh, I’ve just reinvented “days until derailment.” (But worse, since now “days ahead” is undefined / infinite for a flat segment.)
• Can I make flat segments a special case? Maybe, but it’s a bit arbitrary, and you’d prefer for very-shallow-slope segments to work similarly to flat segments. (Although “urgency load but ignoring flat segments” might be an improvement over regular urgency load?)

The more I think about it, though, the more I just want that “non-flat days until derailment” metric: (days until derailment) - (flat days before derailment).

• “Days above” is just “non-flat days until derailment” if a goal’s rate is always r or 0.
• “Non-flat days until derailment” does the right thing with weekends-off goals: “3 non-flat days until derailment” = “3 work days until derailment” no matter what day of the week it is, so you never need to check a calendar to know how to interpret it. Same for vacations, and any other time when rate = 0 implies that you don’t intend to enter data that day.

During a post-derail respite, NFDUD will start at zero and stay there until you start adding data. Some people might want that, but some people might not. I don’t know how I feel about it yet. ¯\_(ツ)_/¯

The effort it takes to increase NFDUD may vary over time if rates change, just like DUD, but I like that NFDUD always measures days. I think some other tool can be used for forecasting.

[hnsn]

I have a branch of the Raycast extension that lets me sort (and color) goals by “days above the line”. The formula I’m using is floor( delta / rate-per-day + 1). It’s perhaps overly simplistic as a formula, but so far I find it completely satisfactory! (If anyone has a better way of calculating this, let me know!)

I’ve also created a new goal for myself that sums the days above the line for the 10 lowest goals, with the goal to get to 100 (i.e., 10 days above the line for my 10 most pressing goals). (I think this is what @scarabaea is doing as well?) Data is added throughout the day via a cron job and a personal API endpoint.

Between these two things, I don’t think I’ve ever had this much buffer! And it feels amazing. I’ve previously tracked total urgency load, but that never really took off like I hoped it would (perhaps for the issues previously stated around breaks and flat spots).

Anyway, please let me know if you have any thoughts or suggestions on this approach! (It’s very possible that I’m missing something important from the discussion above. )

[scarabaea]

@hnsn, yes, this is very similar to what I am doing - the formula looks recognizable as well. I like the sleek look of your list!

[hnsn]

@scarabaea

I like the sleek look of your list!

Huge kudos to @vivgui for the original Raycast extension! It’s my primary Beeminder UI.

[vivgui]

I’m glad you all like it! And same for me, its the primary way I use Beeminder