Revisiting "Red Yesterday"

[zzq]

@dreev

We’re probably going overboard with it right now, but there are advantages.

Yeah, I don’t want to be too harsh about this. The advantages are real. And the disadvantages partially overlap with the existing disadvantages of the database you’re using (MongoDB)—either way, you wouldn’t be getting the full advantages of a well-structured and principled database schema.

In short, I fully agree with you here:

@dreev

But now I’m defending it too hard. With better architecture, better UI organization, less tech debt, etc, I suspect you’d be right that more structured metadata for some of the things we’re using magic strings for (derails, restarts, and, just now, archives) would be better.

I’m not entirely convinced by your quick reactions to the challenges I’ve pointed out about the bright red staircase. I do understand they are just quick reactions; but they are very much ad-hoc, unprincipled. It is in response to proposals like this where the anti-magic principle is at its most necessary, I think. “Conventions” like 12:01 am scare me, especially if that’s going to be baked into code as “magic”.

A truly generalized solution where datapoints may be associated with either points in time or time ranges (representing uncertainty) could work, maybe—but that deserves more thought as to how to do so in a principled, unmagical way.

Also consider a case where a user decides that e.g. they want a daily deadline each day at 3am, except on Wednesdays, where they want hourly deadlines each hour on the hour. How does the interface work for them? Does it change on Thursdays, with “Yesterday” grayed out—or does selecting “Yesterday” work even on Thursdays, somehow? In some principled way, which works not just in this one case, but no matter what? (And what about someone who chooses a continuous road? Some descriptions of the vision of the bright red staircase imply that would eventually a very standard use case. (Maybe even the default for new goals, eventually?))

@dreev

Aggday: I think there’s always a logical translation to staircase world. Aggday=sum is just tracking a cumulative total, which works just as well if no datapoints are on top of each other. Aggday=last, same thing. Aggday=min for weight loss … is the translation here is that the red line wants to be a sawtooth shape that allows a higher weight in the morning or evening?

The notion of aggday is very much dependent on the current (pre-bright-red-staircase) way of thinking about days as a bundle of datapoints to be aggregated. Probably you’d want a different abstraction here entirely. As you say, overall aggregation (not daily aggregation) works equivalently for some, such as sum and last.

The real issue is with aggdays like triangle. (Or square, or sqrt, or mean, truemean, median, mode, etc. Or min or max.) These are explicitly about lumping a discrete collection of datapoints into one. That works well in the current model, and is pretty solidly at odds with a continuous model.

@dreev

Another approach is to say that aggdays apply the same as always, but specifically to points with the exact same timestamp. And then, in the weirder cases where this matters (not do-more or do-less or odometer goals, I don’t think), you can give all your datapoints that you want to be aggregated each day a timestamp of exactly noon today.

Note that this is more than a bit magical, and rather at odds with the most standard use of e.g. triangle for stuff like alcohol consumption—it’s very natural to add a datapoint for each drink you imbibe as you go, which gives them different timestamps. If you have to go back and manually edit stuff to get it to line up… that’s annoying and fiddly, very much dependent on arbitrary and undiscoverable rules (“magic”), and worst of all, it’s fake data. You want a graph that also serves as a Quantifiable Self record of the alcohol you drank, with timestamps—and it’s very much not great if Beeminder makes you falsify the timestamps to meet its arbitrary rules about aggregation.

@dreev

Or we bite the bullet on everything being continuous rather than discrete and accept some tradeoffs. I’m not at all sure which philosophy is best. I do like the Pareto Dominance Principle, but I also like worse-is-better.

Maybe. I think that there’s probably a way to rethink the model of aggregation in a completely different way, from first principles, that fits better into the continuous world.

@dreev

I think the answer to my original question is that checking for derailment is straightforward in staircase world: When we look at the graph at any time t, is there any point earlier than t [optionally: and more recently than a week ago, or whatever statute of limitations] at which the data crossed the red line? If so, the graph is derailed. Compute the most recent intersection of user data and the bright red line and insert the post-derail respite. (See also the sidebar in the uncle button post about post-derail respite in staircase world.)

Yeah, agreed. Consider also a hypothetical “red today” world (which I’m not actually recommending you implement, but imagine for logical completeness)—a very simple derailment model, in today’s discrete world, where if you’re ever in the red, you immediately derail.

That’s almost equivalent to the existing “red yesterday”, just with a different meaning to safety buffer. If you adjusted all graphs to add one day of safety buffer, then switched to the “red today” rule, it would make no difference to most goals. (Except—if you add a negative datapoint putting you in the red, then later the same day a positive datapoint getting you out of the red (or the opposite on a do-less goal) that doesn’t derail you as is, but would in this proposal.)

Why do we have “red yesterday” and not this hypothetical “red today”? I think it’s for historical reasons, an artifact left over after the migration from the yellow brick road model. That’s fine, maybe not worth fixing, but worth thinking about if we’re trying to understand what derailment model we want. If the “red yesterday” rule really is a historical, path-dependent artifact, we probably should put less focus on preserving specifically it.

One could imagine a shift to “red today” as a first step towards the continuous model you described for the staircase world. If so, it would obviate this whole topic of discussion here.