Earlier deadlines unfriendly to QS info (i.e. data is incorrect)

[mary]

I’ve just started using deadlines that are spread throughout the day and I’m liking the way it balances the day out. There’s something that I don’t love, however: anything entered on the 20th after the deadline to get the minimum in ends up posted as though it was entered on the 21st. It makes the datapoint information incorrect. If you’re inclined to export the data and work with it at all, for QS purposes (or even just to create a “how I did each day this month” bar graph), the information you have is wrong.

Random example case: If you want to get info on how hard your crash tends to be after a day where you really push on a given goal, you’re not able to get that information because some/much of the “extra” done for the goal on the bigger days is logged as though it was done on the following days.

More generally: any QS usefulness evaporates.

[mary]

(PS - This probably affects web users much less than API users and other users with any kind of automatic process that uses the “^” to enter the datapoint. For example, simple email- or sms-generating scripts, apps, IFTTT triggers, etc). Though… it’s still likely that even web users wouldn’t (and I sometimes don’t) remember to change the date to “yesterday” when using the web to enter a datapoint for today.)

[philip]

Yes, I collided with that, too!

My chain of scripts driven by (and driving) some of my goals started to get out of whack:

what I have found is that by moving this one goal’s deadline, since I feed it by autodata, I need to move the feeder goal’s deadline, so that they’ve got the same sense of what day it is, which then puts them out of sync with tagtime, both of which feed into my clever spreadsheet that calculates my invoices, which … aaargh!

My current solution (and you may hate this!) is to keep a QS-and-calendar-friendly goal with a midnight deadline, and to mirror all/some of the datapoints to yet another goal that has a different deadline.

---

Would an export calendar-day-datapoints feature be of use?

Related post: One Real World Goal, Multiple Beeminder Goals

[dreev]

We agonized about this a lot when implementing arbitrary deadlines. See section 4 + the appendix of the arbitrary deadlines spec.

Short answer is that the data has the true timestamps but we do treat the deadline as the day boundary for purposes of graphing and displaying. It became a worse-is-better vs the-right-thing debate and I predict @mary will be horrified that we rejected “the right thing” but, well, read about the worse-is-better philosophy before yelling at us too much… (:

PS: Speaking of being horrified, I clicked like on @philip’s reply here since it’s contributing lots to the discussion, as always, but let me disavow the craziness of mirrored goals as a solution to this! (: My first reaction to your “export calendar-day-datapoints” feature was that it’s overkill but if it takes something like that to avoid the contortions you’re going through then it may be the lesser evil…

[mary]

I can accept worse-is-better decisions (in theory, hehe), but in this case I think there might be reason to think that, if it was once better, it might no longer be the case.

Things have changed.
"This is very much what people expect in night-owl mode (cf “it’s still the 30th to me!” and “I want to define when my day ends!”)."
When that was the typical use of deadlines, I could sort of understand it, even if I didn’t prefer it. I mean, I personally still used the actual date, but I got the reasoning behind it. Given the “Chasing Waterfalls” use of deadlines is (I presume) on the rise, though, I think this is no longer clearly better.

Timestamps aren’t all that helpful
It doesn’t matter so much that timestamps remain unaltered, because whether they’re available in the API or not (I haven’t checked, tbh), they don’t export to CSV, and so the real data remains trapped for non-API users. But what’s worse is that sometimes I make a note of something and don’t enter it until the next day if it’s not a potential derailment day (I often do that with sleep, for e.g., taking a screenshot of the time when I shut things down and just entering that time retroactively in the morning – or if I forget to enter something that I’ve noted and enter it the next day, when I see it on my list of things to enter). So working exclusively with timestamps (assuming you’re an API user to begin with) doesn’t work either. I could, if both are exposed in the API, create something that rewrites the date to the timestamp’s date for all datapoints where timestamp < date, while ignoring all datapoints where timestamp > date (to protect the “oops, I forgot” cases), but I think needing to write an API-aided extension just to have the real data is suboptimal.

QS
This doesn’t need an explanation. If the data isn’t accurate, it’s unusable for QS purposes.

Wise people who have put a lot of thought into Beeminder agree
"Beeminder is foremost a Quantified Self tool, so it feels really wrong and counterproductive to falsify your data. People take a lot of pride in their graphs since it’s a meaningful visualization of their progress, and they’d rather pay the occasional penalty than mess that up."
–@dreev

“serious beeminder users know that fake data is anathema to the whole concept”
–@dreev

(muahaha)
(I should add that the last one is taken out of context, and “fake data” is really referring to datapoints you didn’t actually complete)

[dreev]

This is convincing me that, at the very least, @philip is right that the CSV export should include the calendar dates. That might be a nice simple #UVI in fact. (No separate export though, just throw in an additional column.)

@chipmanaged, you are amazing. Thank you so much for the incredibly thoughtful feedback.

Until we get to this, I think the workaround, as you suggested, is to stay hyperaware of the deadline as end-of-day and remember to enter data after it as “yesterday”. Also I just remembered that @philip and I argued about this way back when in the context of the confusion of graphs being red despite not being due till tomorrow. I’ve gotten pretty used to it – red just means <24h – but it’s maybe not ideal.

PS: I’m laughing at the irony of “worse-is-better is better in theory”. I mean, in theory worse is, presumably, worse, but in practice it’s better. Something like that. That was probably your point.

[mary]

I completely forgot about this bug, changed each of my goals to an appropriate deadline for that goal, went through a couple of days of having cascading deadlines and really loved it… and then realized I had to open 75 goals and check to see which had fake dates for the datapoints that were entered after the deadline but before the next day (e.g. if I did extra work on a project after the deadline). There weren’t a lot of them, but I still had to check every datapoint in the last 2 days to see if incorrect data had been entered or not.

What are your thoughts on giving us a “preserve true dates” option (per goal, per user; doesn’t matter to me, but others might feel differently) in the reminders section so I can go back to using this feature plz?

It’s a great idea as a feature and I really enjoyed having it for the (admittedly short) time I tested it out.

[bee]

OK, so first I want to be clear about what is happening with datapoint timestamps / dates when you set a non midnight deadline.

If you give us a timestamp, we don’t mess with it! that includes entering data with a ^ (which you can do on the website in the advanced entry box or the quick add, in emails, and through the api by passing “^” for the timestamp param, or omitting the timestamp param entirely). We also calculate and store the date (which we refer to as daystamp). We calculate the date relative to the deadline.

When you give us a date, we preserve that, ignoring the actual current time. But we also sorta need the timestamp still, so we pick some timestamp that corresponds to the given date, accounting for the deadline and user timezone etc. So, it’s kinda arbitrary, but we settled on picking the last timestamp that corresponds to the given date. The spec maybe says more about why we settled on that. I honestly don’t recall at this point.

So. As to the QS-i-ness of this… we’re making a few assumptions here and there, but we tried to respect the QS-First principle when possible, and preserve the data given to us. As for the CSV export – excellent point that this is the most user-friendly way to get your data, and doesn’t preserve that timestamp info. We did long ago use the timestamps, but then for the sake of user friendliness swapped out for the date. No reason not to include both, I suppose.

The datapoint model also stores the deadline at time of creation, and the tzoffset of the user at the time of creation, so the timestamp -> daystamp calculation should be reproducible.

The weirdest thing happening here is the timestamps we’re generating when you add data using a date (for deadline shifted goals), and I no longer recall the series of events that lead us to that decision. It seemed like the least-worst solution at the time when I knew way more about this stuff.

Bethany

[mary]

This is a whole lot more complicated than I realized (and I suspect many/some people want the behaviour you’ve set up, as it is) so I deem myself unqualified to have further opinions on it!

I’ve just appended the date and time to the comments generated by the API (with a separator, so I’ll be able to use it easily), and I’ll restrict myself to entering data there (at least for the goals where I’ll want to use the QS data further).