Beeminder API: roadall documentation

Tech
1

Would it be possible for the API documentation to include a better writeup of how roadall works, and in particular, how the timestamps work?

I was using Postman to try to manually crank up the road for the next few days for a new goal, and honestly, the process was extremely frustrating. I couldn’t seem to get anything to match up (you can see what I was actually dealing with below).

There were a few thing in particular that didn’t seem to make sense:

  • It’s unclear why all of the datapoints are dated at 16:00 GMT on any particular day. When I originally tried to use a timestamp that was after the time that I was submitting it but before 16:00 GMT, it derailed my goal, so the significance of this timestamp is confusing.

  • When submitting timestamps, I kept ending up with very unexpected results. I created the goal with no buffer time and it said I had 2 days and some hours before derailment. I retroratched to 0 days, and that put my goal for today on the “Amounts Due by Day” to -2 for today. As I increased the amount that was due in subsequent days, the amount that was due by today actually decreased.

  • Why doesn’t the delta_text field returned by the GET endpoint match up with what’s in the “Amounts Due by Day” box? According to this box, the deltas for today and the following two days are +0, +8, +15, yet the delta text field states +0, +8, +16.

  • What’s the cut off for rate changes? “Take a Break” states that the rate should increase by 3 “Until 2016-04-02,” but the amount due on 2016-04-02 is 18 and on 2016-04-02 it is 21. It seems that the rate changes on the day following the end date of the rate.

  • In addition to the previous point, there is something that is completely baffling to me:

    • fullroad shows the list [1459612800, 21, 3] (timestamp: Sat, 02 Apr 2016 16:00:00 GMT)
    • “Take a Break” shows Until 2016-04-02 increase at rate 3
    • “Amount Due by Day” shows Sun(3rd) +21 21
    • This means that there is a datapoint which explicitly states that at Sat, 02 Apr 2016 16:00:00 GMT, the amount due should be 21 and the rate at that time should be 3. However, elsewhere the interface is showing that 21 are due on the 3rd, which is 3 more than the amount due on the previous day, the day that the rate should have stopped increasing.

I have to be missing something there… how is the road actually calculated? Why does it seem like none of the numbers are lining up?

tl;dr: help

Under “Take a Break” I have this:

By 2016-03-31 reach value 15
Until 2016-04-02 increase at rate 3
Until 2026-09-28 increase at rate 2

If I get the goal, my roadall reads:

  "roadall": [
    [
      1459353600, //Wed, 30 Mar 2016 16:00:00 GMT
      0,
      null
    ],
    [
      1459353600, //Wed, 30 Mar 2016 16:00:00 GMT
      8,
      null
    ],
    [
      1459440000, //Thu, 31 Mar 2016 16:00:00 GMT
      15,
      null
    ],
    [
      1459612800, //Sat, 02 Apr 2016 16:00:00 GMT
      null,
      3
    ],
    [
      1790611200, //Mon, 28 Sep 2026 16:00:00 GMT
      null,
      2
    ]
  ]

If I look at the “Amounts Due by Day” I see:

DAY		DELTA	TOTAL
Today		✔	0
Tomorrow	+8	8
Fri (1st)	+15	15
Sat (2nd)	+18	18
Sun (3rd)	+21	21
Mon (4th)	+23	23
Tue (5th)	+25	25

And the get GOAL API endpoint returns:

"delta_text": "+0 +8 +16",

3 Likes
2

I’ve had the same problems but I never was able to figure it out. I gave up on that particular project. I say this not to discourage you, but to at least validate that you are not crazy: https://xkcd.com/979/

2 Likes
3

There is a lot of cruft returned by the API,
Roadall is the only trustworthy thing.

Also: The road doesn’t specify the bottom edge of the road, but rather the center, with the bottom edge being decided by a confusing ‘lanewidth’ algorithm.

Hint: set your ‘lanewidth’ to 0, this loses you any warning you’re going to derail but you can be sure where the edge of the road is.

1 Like
4

This is a mess and is important! Thank you so much for asking these questions. @bee and i are going to answer them in the next 30 hours…

5

Ah, that explains the 15/16 discrepancy.

1 Like
6
  1. we need better documentation surrounding road manipulation. Things like the road width gotcha that insti mentioned, and careful description of the [date, value, rate] triples – namely that the next segment begins the day after, so a date,rate pair means “go at Rate until (and including) Date”.

The right thing to do is for us to be using daystamps, because everything is dayfloored in beeminder in terms of times and that’s what we really mean anyway. It would be so much more straightforward and route neatly around deadline related issues and I’m thinking that rather than answer any of your detailed and specific questions, I’d do better to just implement an api v1.5 that changes the road stuff to use daystamps. If I only implemented a GET and a POST on individual goals, it wouldn’t be a huge undertaking, and then everyone would have a sane interface to roadall and be able to actually build stuff on it instead of abandoning projects and other sadnesses. :cry:

Ok, but since I haven’t actually done that yet, the timestamps in the road are all for beebrain (the graph generator)'s consumption, and so they’re all at noon EST, or 4pm UTC, so if you convert them all to dates using that info, then maybe you can get it to work as is in v1?

So the following road:

[
  [1455642000, 0, nil], 
  [1455642000, 0.0, nil], 
  [1455728400.0, nil, 0.0], 
  [1502380800, nil, 1.0]
]
# should be thought of as:
[
  ["2016-02-16", 0, nil], 
  ["2016-02-16", 0.0, nil], 
  ["2016-02-17", nil, 0.0], 
  ["2017-08-10", nil, 1.0]
]
2 Likes
7

This clears things up a lot, but I’m still confused a bit about the “until and including” part.

In the example road above I had

    [
      1459440000, //Thu, 31 Mar 2016 16:00:00 GMT
      15,
      null
    ],
    [
      1459612800, //Sat, 02 Apr 2016 16:00:00 GMT
      null,
      3
    ],

So it seems up to and including would mean that the rate would keep increasing by 3, even on 2016-04-02; this would result in a target on 18 hours by the deadline on the 2nd. But the amount due by day also showed Sun(3rd) +21 21. I think I’m just being confused about phrasing but I would think “up to and including” would mean that the rate increases by that much until the deadline of that night.

Do you mean that the value for a day is determined by how much it increases the previous day?

8

Ok, so I’m pretty sure that what you’re seeing with the due by table is what insti tried to describe above about road width changing. Let me give an example to make it more clear:

In this first road, I’ve got a flat spot scheduled to start today, where the road is set to 0 for the two weekend days. The road matrix looks like this:

[[2016-04-02, 0, null],
 [2016-04-08, null, 7], 
 [2016-04-10, null, 0],
 [2016-04-30, null, 7]]

The road produced looks like this, and the due by table follows.

I’ve still got one due today, because I’m in the red still, and then 0 additional due for saturday and sunday (the 10th). So the road is continuing flat at a rate of 0 until the 10th, and then beginning on the 11th it begins to rise again at a rate of 7/week.

Ok, second example. Similar deal, but instead of scheduling a 0-rate flat spot, I just decided to dial down to half my usual rate over the weekend. The road matrix looks like this:

[[2016-04-02, 0, null],
 [2016-04-08, null, 7], 
 [2016-04-10, null, 3.5],
 [2016-04-30, null, 7]]

Look at the Due By table in this case however. I’ve got 1 due today, to get back on the road, and then my road rate changes for tomorrow and so you’d expect I don’t have to do an entire +1 again tomorrow, but in fact I do because as well as the road changing, the road thickness changes.

There’s also some confusion with what the Due By table is showing, because I’d expect it to be showing me that only 0.5 is due for the sunday, but in fact it isn’t. @dreev could maybe explain that better than me.

1 Like
9

I monotonized it. It was confusing/misleading for it to say you have +1 due tomorrow and then only +.5 due the next day. If there’s +1 due tomorrow then there must be at least +1 due every subsequent day.

Of course this is all horrible kludgery until we rearchitect the yellow brick road to be based on the critical edge and not the centerline. That will need to wait till version 2 of the API though. We’re codenaming that project “yellow brick halfplane”. HT @insti for making the case for it!

10

Hey @nickanderegg and @drtall and @insti, thanks to @apb you can now specify “daystamp” strings like “2016-05-25” instead of timestamps with byzantine timezone rules! It’s not documented yet and the api will still return timestamps which makes it confusing but we’re eager to hear if it’s a step in the right direction. We’re thinking that in v2 of the api everything will be given and returned as daystamps.

Actually, @apb or @bee, could we provide (here for now) the functions we use internally to convert back and forth from daystamps to timestamps?

2 Likes
11

Cool! Should we send “measured_at” as well?

12

Apologies – I’m confused about the question here, @ianminds. Measured_at is a datapoint field, dreev was talking about times/dates provided to road_all.

And here, per @dreev’s request, is the code for converting between timestamps and dates that we use in road_all, though, recall that in roads part of the whole reason this was so confusing to begin with was that we ignore user timezones and always store (and assume) US Eastern time for timestamps:

# Date => timestamp 
daysnap(date).to_i

# Timestamp => dates
Time.zone.at(ts).to_date       # default zone in beeminder is set to US Eastern

# :sec seconds of offset to add, by default 12 hours, e.g., noon
def daysnap(time, sec = 12.hours)
  if time.is_a? Numeric
    daysnap(Time.zone.at(time),sec.seconds).to_i
  else
    time.to_date + sec.seconds
  end
end

The daysnap function relies on rails magic (I think from ActiveSupport). When we add 12.hours to the Date, rails magically casts it back to a Time using the default timezone (in the Beeminder codebase this is US Eastern time).

2 Likes