Visualizing beeminder data

I think another way to make it a bit simpler to contribute for everyone (and fix Firefox issues) is to use Google Spreadsheets.
We’d need a function to fetch data from beeminder and then everyone can build charts using Spreadsheets UI.

You can do it using: https://raw.githubusercontent.com/bradjasper/ImportJSON/master/ImportJSON.gs - copy this script to Apps Script in your spreadsheet, save it. Then you’ll be able to import goal data:

=ImportJSON("https://www.beeminder.com/api/v1/users/skorytnicki...")

and you’ll see your data in the Sheet:

From here you can use charts provided by Google and their community’s plugins.

Thank you good people of beeminder for providing API endpoints!

1 Like

Here’s my version of @skorytnicki’s original. Beeminder visualization tools / Kirsty Darbyshire / Observable

This one:

  • sums multiple data points on a day
  • works around a problem where summer daylight saving datapoints appear on the previous day - there are no doubt better ways to do that!
  • calculates the starting date for the calendar from the datapoints themselves

I think it will have created problems if your datapoints aren’t in ascending order (by day at least) though, since it presumes they are in order to add up the multiple points on a single day. Adding a sort would help, but it works well enough for me for now so I’m going to go and add some more datapoints to things rather than fiddle with it more :smiley:

3 Likes

You wouldn’t believe what I just found; a beeswarm chart!

You’ll find excellent beeswarm charts around there on Observable. They’re amazing. This one is pretty simple and shows at what hour you performed on what goal. Each datapoint is a small dot. This way we could’ve seen how we accumulate small dots over time; their color will tell us what sort of work we did.

From there, it’s takes some JavaScript fu to make it show your busiest month: moment(d.updated_at*1000).format('MM'), week or day of the week.

Also I want to point your attention to two tiny blocks:

viewof showSecret = Inputs.checkbox(["Yes"], {label: "Show secret goals"}) and then: viewof selectedGoals = Inputs.checkbox(goals.filter(goal => showSecret.length === 0 ? !goal.secret : true).map(goal => goal.slug), {label: "Select goal", value: 'commits'})- I filter non-public goals out by default.

Another one is this utility to convert some text to color: stringToColour which is the snippet I copied well over 9000 times. It allows you to generate the colour from some random text. stringToColor('duolingo').


I still chew on what @philip showed me about cumulative graphs. Thank you Philip for keeping the thread spinning.

I think there’s strong notion of journey (road; yellow thingy road) in founding team somewhere, which is I guess correct mental model to visualize Beeminder data. There’s something precious behind you, and something exciting ahead of you. Successful todo apps (beeminder, things3) are time-oriented, so is beeminder; beeminder takes it to the extreme since each task is assigned a perfect point in time to happen. Hence some sort of timeline would be a good way to visualise the progress. Something like this might work for me:

This could be horizontal or vertical, preferably scrollable or zoomable. There could be squares or triangles if you are circlephobic. This chart takes all (or subset) of your goals, but it is not important; it can be a per-goal chart. Pay attention to the big picture, please (though I posted a small one lol).

I see some cons of this chart:

  • how about fractional beeminding?
  • canonical chart has lots of nice metadata (polynomial fit etc)
  • it works nicer for goals with multiple entries a day; would look pretty sad for my goals (1 thing / day and like 3 things a week)
  • rendering corner cases especially with multiple goals where value is not interchangeable (1000 lines of code vs 20 duolingo points vs 1 toothbrushing). This could be solved by using color instead of size to express the velocity. Or shrinking the past to make it look more busy.
  • does not work fine when rendered on small screen (imagine 100x100px chart).

Note that this chart is not much different from canonical chart. We simply make it flat. The velocity is expressed via bubble size, not Y axis (Y axis is gone, I had to put their data somewhere).
However, my chart emphasizes good things ahead of you (yay clean teeth), while default chart makes you scared about the collision with the red line.


It is also a convoluted reply to the most recent blogpost where we’re given heads-up on the red line turning into red staircase. I appreciate the iterative approach we have there, but I wonder if we were to design it from scratch, would it still be the most informative design? What were the other options? I guess you had this discussion before though.

2 Likes

@dreev suggested I ask here: anyone have some easy trick to plot the Bright Red Line aside arbitrary data? In my case, I wanted to see the ground truth weight from my scale overlaid against the Beeminder view (Beeminder only plots the point for the daily minimum).

2 Likes

I just realized that maybe all you’re looking for is the “plot all” setting?

PS: Everyone contributing to this thread is amazing and if you want some premium credit with which to experiment with “plot all” or anything else beeminder-data-visualization-related, DM me and it’s yours.

3 Likes

I have no clue unfortunately. API provides all data we need regarding red line, but I haven’t played with it yet.

Nonetheless we are here for fancy viz. I am here only for 3 months now, but it’s enough to show at least some patterns.

Here’s everything I did with beeminder so far. This would not happen without beeminder, at least not with such beautiful cadence and regularity that build up to better life.

Some days are more busy than the others, you can also see that I started to measure more things over time. I think I’m done inventing new goals for now.

You can also see focus-time activities (i.e. after work, kid sleeps etc):

vs coffee time activities:

Also Anki and Duolingo are small. I can do them every day (0 - Sunday, 6 - Saturday):

while Fridays are less productive in terms of coding. Whole week seems to take its toll:

while reading books makes me apparently more relaxed:

Also it’s hard to focus on what is very demanding (being creative, i.e. coding):

while small or relaxing habits like duolingo or reading books are easier to start and keep:

The point of visualizing the data is to take action. So I can now clearly tell that between 6 am and 6 pm I work for people (family, job; don’t get me wrong, I love both) while past 6pm I start to reclaim my focus time. The goal now could be to pull the rope harder and start living my own selfish life earlier; make the day more balanced at least. I could set a goal to “reorganize your life so that you can code and read at 2pm”.
Also since data shows it’s very easy to pick up duolingo and anki - maybe I could start more things like that. Or not, maybe they’re parasites and just useless tokens of activity to make me feel better, but you can’t tell it from the graph :slight_smile:
I hope I would be able to reverse engineer changes in my life over the years if I stick to beeminder; e.g. is new house (more comfortable) buying me more spare time or is the new kid taking it away from me. Are older kids more self-sufficient so that I can read more or nope? Interesting times we live in.

2 Likes

That’s fun. Here you can see when I am usually meditating. The dots go pretty high up so it would make sense to normalize them in my case.

I would find a Beeswarm graph over time useful. I like to divide my goals into the areas Mind/Wealth, Heart/Social, Body/Health, and Soul/Spiritual. So my idea would be to have one circle for each category, and then have a timeline as you have sketched out.

The size of the circle would indicate how much effort I have put into a specific area on a given day.

A feature request: could you change the calendar view template so that one can select archived goals? I would like to see how I performed for some bigger projects that I did using Beeminder.

1 Like

What I’ve done is put these three lines in place of the single line to get the goals:

code to include archived goals as well
archivedgoals = 
await fetch(`https://www.beeminder.com/api/v1/users/${BEEMINDER_USER}/goals/archived.json?auth_token=${BEEMINDER_AUTHTOKEN}`
    ).then(r => r.json()).then(json => json.map(goal => goal.slug))

currentgoals = 
await fetch(`https://www.beeminder.com/api/v1/users/${BEEMINDER_USER}/goals.json?auth_token=${BEEMINDER_AUTHTOKEN}`
    ).then(r => r.json()).then(json => json.map(goal => goal.slug))

goals = currentgoals.concat(archivedgoals)

2 Likes

@felixm

The size of the circle would indicate how much effort I have put into a specific area on a given day.

In fact beeswarm charts are not much different from my previous charts: Beeminder and time / szymonkorytnicki / Observable - maybe these would work for you? But for each goal / group of goals you’d have to have a separate chart.

I recently saw Beeswarm Mirrored chart, that aligns things a bit nicer:

import {BeeswarmChart} from "@d3/beeswarm-mirrored"

Here’s my progress on Beeminder so far, looks pretty sweet.

Then I also exported my Observable dashboard as HTML and uploaded it on my server. Then, I opened the page in Safari and pinned as an application on my iPhone. This way you can substitute regular app with your own app. This exporting/uploading dance is necessary to get rid of Observable’s top bar taking away too much of screen real estate.

Remember; never ever share exported HTML as it contains your private data.
Also, if the export doesn’t work, substitute Secret() calls with your beeminder username and password.

1 Like

Nothing interesting from me today. From time to time I work on my dashboard. I want to have nicer overview:

boxes:

html`<div class="goals">${goals.map(goal => `<div class='goal goal-${goal.slug} goal-${goal.roadstatuscolor}'>${goal.slug} <div class='footer'>${goal.limsum}</div>${goal.todayta ? "<div class='dot'></div>" : ""}</div>`)}
<style>
.goals {display: flex; font-family: -apple-system, system-ui; font-weight: bold;
    align-items: center;
    flex-wrap: wrap;}
.goal {position:relative; padding: 10px; box-sizing:border-box; border-radius: 5px; background-color: gray; color: white; width: calc(50% - 10px); text-align: center; margin: 5px; text-overflow:ellipsis; overflow:hidden; white-space: nowrap;}
.footer {font-size: 10px; font-weight: 400}
.goal-green {background: green}
.goal-blue {background: blue}
.goal-orange {background: orange}
.goal-red {background: red}
.dot {position: absolute; top:3px; right:3px; width: 7px; height: 7px; background: rgba(255,255,255,1); border-radius:50%;} 
</style>
`

sparkline:

{
  goals.forEach(goal => {
    const $el = document.querySelector('.goal-'+goal.slug);
    $el.appendChild(sparkline(goal.recent_data.reverse().map((goal, index, arr) => goal.value + (goal.index > 0 ? arr[index-1].value : 0))));
  })
}

sparkline function:

function sparkline(values, width = 64, height = 17) {
  const x = d3.scaleLinear().domain([0, values.length - 1]).range([0.5, width - 0.5]);
  const y = d3.scaleLinear().domain(d3.extent(values)).range([height - 0.5, 0.5]);
  const context = DOM.context2d(width, height);
  const line = d3.line().x((d, i) => x(i)).y(y).context(context);
  context.strokeStyle = 'white';
  context.beginPath(), line(values), context.stroke();
  return context.canvas;
}

I think we’re able to recreate the whole graph, this would be cool.

White dot in top right corner suggests if goal was done today.

2 Likes

Hi all,

I started to develop an app as it’s much faster and more flexible than Observable for me. Working with Beeminder API is pleasure and I can focus only on new features and dataviz.

I slowly migrate things one by one. My MVP is to have goals list, goal page with graph, recent datapoints, basic meta data and that cool calendar graph from the first post. Also, I want to use some nice modern UI components like Tailwind or Ant.

Right now you can build and host it on your own. Very unfriendly for non-technical users. With enough momentum though (like 10 asks?) I could deploy some multi-user version (i.e. you can log in and see your profile right away like you do with beeminder, facebook or whatever).

If by any chance you’re interested in contributing, feel free to issue PR, fork it etc.

Basic first iteration; proof it’s totally possible to have alternative beeminder app with such a great API:

I’ll come back in few weeks with stuff that is “done” for me :slight_smile:

3 Likes

And a question from my side. Do we have some list of past ideas about how to visualise The Graph? I know the current thing is replacing red line with red brick road. What are the other alternatives?

I think default graph for my app would be something simpler than The Graph; I want it to look very basic and minimalistic, like this: https://charts.ant.design/en/examples/tiny/tiny-area#basic-area - no X, Y lines, datapoints labels visible only on hover, just two lines (red line and blue line of your datapoints).

I imagine it could also look like vertical bar chart (day by day); width of the bar is a distance from red line.

cc @dreev @philip you probably can point me to some topics or insider knowledge; I remember this: Why Beeminder likes cumulative graphs

Awesome work! And thanks for the kind words about the API. I sadly predict you’ll run into frustrations at some point – which we definitely want to hear about! We have a long list of things we want to do better in the eventual v2 of the API.

Other way around! Goodbye Yellow Brick Road, Hello Bright Red Line | Beeminder Blog

Also I should mention that Beeminder’s graphing is all open source: GitHub - beeminder/road: Beebrain and Visual Graph Editor

Ah sorry; I meant that: The Bright Red Staircase | Beeminder Blog

Alright, enough buildup, “bright red staircase” means that the bright red line that Beeminder draws on your graph — the line you must keep all your datapoints on the right side of [1] — should reflect exactly what the deadlines are. If there’s a deadline every day — as in the current Beeminder — then we’d draw the red line not as a smooth slope but with 24-hour-long horizontal segments punctuated by vertical jumps at your daily deadline.

Thanks for sharing the graph repo!

1 Like

I keep journal of my app development on Twitter, for anyone who’s interested in such details. I keep backlog on Github if you’d like to contribute or fork for your own needs.
This is enthusiasm-driven development - I don’t have any particular agenda, just pick up whatever I find interesting at the moment. You can influence this process.

I have some new charts. This is datapoints + their weight. The longer run, the bigger circle:

Note we can apply different kinds of regressions (linear, exp, loess, log, poly, pow, quad).

I use Ant Charts for my app: https://charts.ant.design/en/examples/scatter/bubble#quadrant-tooltip - for observable you can use https://g2plot.antv.vision/en

Once again, the more interesting data, the better chart gets. Instead of tracking times you run, it’s better to track kilometres, instead of number of commits, track lines modified, instead of reading sessions, it’s much more interesting to plot pages read :slight_smile: Mine are pretty boring :confused:

I think I might add new goals on top of existing ones, just to get these beautiful charts (so keep reads | skorytnicki/reads goal page aka “has read anything today”) but add a goal with pages read :slight_smile: runs-10k | skorytnicki/runs-10k goal page is the experiment for that.

I can use color, shape or anything to convey some meaning.

red circle - before 12:00, blue circle - after 12:00.

Re: my previous point on interesting charts. Reporting info on each cup of water drank is an obvious idea, but it was so annoying :slight_smile:

Edit 2: Another option, less flexible though, is to store some meta data in the comment; I think it was mentioned already in this thread.

I saw great “trend” feature in Apple Health recently and decided to implement it in my app. I should’ve figured that out in my previous post. These are my recent runs and I’m trending to run more kilometers :slight_smile:

Also I have love hate relationship with Seinfeld’s Chain. I added “current streak” tile to help me grind Duolingo. I have no problems with Anki though:

“Trends” chart sometimes doesn’t work; imagine goal like “has commit on github”, essentially true or false, 0 or 1. Trend here is that I always insert 1, when I derail, I will do 0.

I added new type of chart where I aggregate values by week:

It is still interesting for properly measured goals, like kilometres ran per week:

Strava provides very similar chart in their app and we will have it too. Y axis is sum of values for a given week. X axis is time. Red line could be median or average, I haven’t decided yet on what to use; I like how Apple Health handles it and shows that the recent weeks/days were better than the previous.

I think you liked Observable where you could play with particular charts; I am slowly getting to the point where the app is ready for you and then you will be able to enjoy it without coding. For those who can read JavaScript, there’s a repo of this app linked in this thread, and anyway it’s just Ant Chats you can use with Observable or any other playground.