An experiment to beemind "keeping on top of grading"

We’ll see if this actually works, but I’ve just set up a goal along with my newly-received first stack of grading for the term, and I’m excited.

The metric I actually want to reduce, I’ve decided, is “horribleness of my future grading experience if I don’t work on this now.” Which can be numerically expressed as “the average hours per day I would have to grade to finish my current backlog on time.”

I actually do calculate this number already, because I find fiddling with charts a highly motivating way to make grading more appealing. So this is what I set up for my grading spreadsheet:

As you can see, it’s a piddling little pile right now, except that if I leave it all to do the day before, I will have a very unpleasant day. (The “hours” calculated from the number of assignments is based on the time per assignment allotted in my contract, which I stick to scrupulously, so it actually is a predictable time commitment if I have a count for the number of papers.)

The goal I set up looks like this:

Set-up notes:

  • Do-less goal EDIT: now using whittle-down
  • Flat slope, with the “value” set at the maximum backlog that is acceptable to me (currently trying 3 hours)
  • Data aggregation type “min” (EDIT: this is the default with whittle-down)
  • Presumptive pessimistic datapoints on EDIT: because whittle-down goals can’t have PPDs, I’ve hacked one together with a Zapier zap that inputs a datapoint of 4 every day. I have to edit the datapoint with the correct backlog every day to avoid derailing. (Note that the datapoint has to be above the maximum to trigger an eep day; if the fake datapoint is exactly 3, with my set-up, it just puts me in the blue.)

If my test datapoint of “5” is the only one in there, the goal goes red and I have until the end of the night to get it back under my limit. When I put in my real current backlog I’m in the green.

What I hope will happen is that I will manually input every day the average hours remaining, which will increase every day unless I grade. I think this will help make sure the backlog stays in my mind as something that “grows” over time rather than staying stable. It also gives me a reward for grading early. If there’s no grading to do, I have 0 average hours remaining! So it seems like this can be flexible to the actual amount of grading on my plate.

The only thing that concerns me right now is making sure this goal will get me in trouble if I ignore it, since I have to manually enter data, but hopefully presumptive pessimistic datapoints will work for that!

6 Likes

Hmmmm, today’s datapoint did not produce the expected behaviour, so now I have started one as a Whittle-Down goal.

That tentatively looks better, except now I don’t have presumptive pessimistic datapoints. Maybe I need to have Zapier put in a datapoint of 3 every day, that I manually update with whatever the backlog really is?

1 Like

Oh, I like this metric a lot! This is something I have been trying to solve for a while too. I will definitely be interested to hear about the results.

1 Like

Today I learned that my homegrown “presumptive pessimistic datapoint” has to be above my maximum, not at it, or I just end up in the blue. But otherwise, the simple Zapier integration is working, and this configuration does seem to be tracking this metric accurately! I once again did no grading, and accordingly increased my average daily grading remaining.

1 Like

I decided to set my deadline a little earlier for the arrived-on-time assignments, to clear them out of the way before a trip I’m going on this weekend, and I’ve been finding this goal very effective! It was extremely flexible for my change in plans – I just changed how I calculated my avg hrs/day grading backlog, didn’t have to change the goal itself at all – and although the backlog has been growing, it’s growing at a much, much slower rate than it usually does for me. I actually look at my grading spreadsheet every day, and feel like I’m doing my future-self a real favour any time I sit down and just whip out one paper between meetings.

10%20AM
24%20AM

I don’t think I’ll know for sure how well this works until the end of the semester, but it’s going well so far.

2 Likes

This looks great and like it will make your grading life so much more pleasant!

It’s similar to what I’m doing with some projects that are tracked in OmniPlan, actually, and where it’s the current overdue hours for the whole project that are tracked and whittled down to 0 as the project deadline approaches, so that I don’t have to make up the whole backlog in that one final day, but bit by bit over time. (I use IFTTT for faux-pessimistic-presumptive datapoints on a bunch of whittle-down goals too, and I find this is really helpful in remembering to enter datapoints every day.)

Brilliant, and I think I’ve just decided to make it a rule for myself to do this as well. Thanks for this idea.

1 Like

For the very first time, I finished all of my grading WITHOUT having to do something that felt like a soul-crushing grading binge!!! I did have to grade at a Greyhound station at 3am, but that was unrelated to akrasia.

I felt that the system adapted really well to the change in backlog when I decided that I wasn’t going to be able to finish everything before leaving on my trip and therefore pushing back my deadline. Rather than feeling a false sense of “deadline is gone!!” relief, it was foregrounded to me that I still had to make some progress, just less.

I definitely notice that the backlog increases more steeply than I emotionally expect it to. Sometimes skipping a day barely seems to make a difference, but as there are fewer and fewer days remaining each skip increases the backlog dramatically more. This makes sense when I think about it, but it does sometimes mean that I was surprised by the impact of a day off. This uneven impact of skipping a day might even be the core akrasia problems: one day off doesn’t really matter, but I can’t take every day off. I think I will get used to this over time.

I now just have a few papers in my to-do pile, which arrived late and therefore are returned to students late, and I am feeling really good about the next batch!


2 Likes

This seems like a good place to note that if the comment on your homegrown pessimistic datapoint starts with “PESSIMISTIC PRESUMPT”, it’ll be automagically deleted when data for that day is added. (i.e. it matches the regex /^PESSIMISTIC PRESUMPT/)

2 Likes

This feature doesn’t seem to be working – my datapoints with “PESSIMISTIC PRESUMPTIVE DATA” are all still present – but I’ve stopped bothering to delete them since the red dots don’t “show” once the real data is added anyway.

1 Like

So, I’ve discovered a flaw in my system: what happens when it’s after my grading deadline and I still have a few papers left to do?

Earlier when I missed my self-imposed deadlines, the backlog was large enough that I’d derail if I buried my head in the sand – which meant that the goal had the intended effect of making me say to myself, several days before a grading binge, “this is unrealistic and I need to make a new plan.” I swapped in the hard deadline for the self-imposed early deadline and kept working away at the new, slightly more manageable pace (instead of either ignoring or bingeing the backlog in a procrastinating panic, as I would usually do!)

But then I got down to just 3 papers left, the ones that had arrived late in the first place and therefore had more latitude in when I had to return them, and… I just sat there with a backlog of .8 for days.

39%20PM

On the one hand, perhaps this was fine, since it was reading break and my real, real hard deadline is the class tomorrow where I see these students face to face. (I’ll meet this deadline no problem.) In which case I should have updated the due date and been recording backlogs of, like, .05 for a bit. And since there was only about 45 minutes to do, I always felt like I it would be a breeze to knock it all out in one sitting if I needed to.

On the other hand, I did promise the students to have these papers back via email before this point, and my union contract says they were due Nov 7, so it was a pretty real deadline, and I do in theory want to… meet my deadlines.

I think my conclusion is – the goal here is not to end up binge-grading. I didn’t end up binge-grading. Therefore, this behaviour with overdue work is ok. If I want to prevent grading from being overdue I need to beemind overdue-ness directly.

2 Likes

Hmmm… maybe it’s tied to goal type.

For myself, I’ve noticed that updating an existing datapoint (i.e. fixing the date) doesn’t trigger deletion of the presumptive datapoint either.

1 Like

Thanks again for sharing your experiment! You inspired me to create something similar and I think the metric — “hours per day needed to finish current backlog on time” — is absolutely brilliant. It:

  • Automatically adjusts to the actual amount of grading I have to do, rather than just e.g. requiring a set amount of grading per day (which I’ve tried but doesn’t work well at all)
  • Actually incentivizes me to get real grading done, rather than letting me get away with spending time doing less-productive-yet-related things such as fiddling with a rubric
  • Indirectly encourages me to get better at accurately estimating/planning how long I will spend grading/providing feedback on individual items
  • Doesn’t simply jump through the roof the instant a big assignment is turned in

The downside of course is that it requires some spreadsheet- and IFTTT-fu to set it up in the first place but I don’t mind that, in fact I really like having a spreadsheet of all the grading I have to do, which I’d never bothered to make before.

I think coming up with a reasonable metric which actually measured something realistic and created the right incentives was THE big problem in Beeminding grading, and based on my experience so far I think you have pretty much solved it.

Here is me actually Getting Stuff Done! :

image

I’m on track to catch up on everything by next week and the end of the semester suddenly feels totally manageable instead of overwhelming.

4 Likes

I’m so glad it’s working for you, @byorgey!! It intrigues me that you have a downward slope rather than a flat one – are you grading pure hours of grading remaining? Surely you didn’t start with an average 20 hrs/day required to finish your backlog??

I am several hours in to a hideous grading binge and feeling less delighted by myself, but, well… that’s what “derailing on this goal” looks like, I suppose.

I decided not to do any grading while I had friends visiting; eliminating the days that they were in town as viable “grading days” produced the very high avg hrs/day number that my backlog sat at while they were here and I was not grading. And then they left town and I… did not grade 2.6 hours on my first newly-eligible grading day, with the predictable result that I had a truly enormous backlog the next day. Continuing to not really grade much led to an appropriate derailment.

I’ll get this done in time in the end, but it’s made me think that 3 hours might be too high a default cap for my backlog… it’s really hard to sit down and grade for 3 hours a day several days in a row at crunch time. You can see that I’ve fiddled with the road. At the end of term I know I have to turn around all the final exams in 4 days (well, or grade on Christmas) so I adjusted the road to allow a 4hr backlog once the exams come in, but I’m going to try to whittle down the final essays backlog ahead of then.

1 Like

Well, the way I set up my spreadsheet, when something has been turned in more than 7 days ago, then the “number of hours per day” needed to get it done “on time” is just equal to the number of hours needed to get it done. (I thought about having the number continue to go up once something is late, to really encourage me to get late things done, but decided against it, since it seemed rather artificial. I could imagine a scenario where there is some small yet distasteful thing which I let linger forever because as long as I get other things done, it won’t by itself put me over the limit; if that really becomes a problem then I’ll have to make another goal for that.)

So yes, I actually started with 20 total hours of overdue grading (!) that I’ve been whittling away at. I don’t have any other external motivators such as union contracts getting me to do this — the 7-day turnaround time is completely self-imposed in my case — so I had just kept letting things slide and feeling vaguely guilty about it.

A limit of 3 hours a day also seemed like a lot to me. For next semester I think I will probably set my limit to 1 hour/day. Though what is reasonable depends a lot on how much total grading you will have to do, how much could come in at once, and so on.

1 Like

There’s a good lesson in here. Often we need to beemind real-world goals from several directions at the same time.

On the plus side, that means that the beeminding from any one direction can be relatively lenient, because it’s the combined effect that has real value.

2 Likes

Progress report:

This is still working really well for me. A few big things just got turned in (e.g. a final exam that I estimate will take me 6.5 hours to grade) and you can see the data points starting to creep up but I am able and motivated to stay on top of the goal to keep me on track. I am all caught up at this point (i.e. the only outstanding things were turned in < 7 days ago), and although I have a lot of grading still ahead of me (10 hours outstanding right now, plus another 4 hours or so that’s going to hit on Monday), but I am not stressed about it in the least. (Well, not much.)

2 Likes

This is pretty sophisticated stuff. I’ve been doing my grading-tracking by counting the number of assignments to mark, setting a whittle-down to 0 on the deadline and retroratcheting away all the safety buffers so I have to Start Today. This works, but beeminder gets a bit confused when you ride on the edge so much, and i sometimes have to contact support to get things set up/shut down. Maybe I’ll try this next time.

1 Like

Indeed, it is a bit sophisticated getting it set up. I’d be happy to share my spreadsheet with anyone who’d like to try something similar.

1 Like

Wow, it’s been two whole years since I started using this system, and it’s still working well for me! I can’t remember the last time I had to spend a whole day binge-grading, and I even had a friend ask me whether I was still teaching because it had been so long since I complained about grading. Two additional lessons learned from the last two years:

  1. The chart has to occasionally go down to “0” to force the last few things to be completed.

  2. Grading in little bites instead of binge-grading requires some adjustment of technique, since it will no longer be possible to mentally “hold” all assignments in my head at once to make sure they’re graded accurately relative to each other. Adopting these new techniques further reinforces the shift away from binge-grading.

The main technique I’ve found crucial is using rubrics more, and making those rubrics much more detailed, perhaps several sentences for each quality level of each metric. When I binge-graded essays I’d skim them all, arrange them in order of quality, identify the cutoffs between the letter grades, and then assign percentages in chunks. Writing a detailed rubric requires putting in more work up-front to define those letter grade cutoff points as abstract principles, but then those descriptions make it possible to assess essays consistently without comparing every essay to every other essay. It seems obvious when I write it down, but it really was a necessary workflow shift to get away from the binges! (I suspect students find the more precise guidance useful, too, especially when the assignment design is unconventional in some way.)

As a closing brag, here you can see that all last semester I avoided having a backlog of more than an hour:


You can also see that the end-of-term spike came a week later than I’d predicted (since the whole course got delayed a week by COVID), so I had to do some edge-skating, but it was still much less stressful than my usual binge.

3 Likes

Oh, I’ve also somewhat refined my grading-tracking spreadsheet:


This is a much more compact spreadsheet – one row per assignment, rather than each assignment having its own little table, with a “master” table summing them all up. I also specify in a column how long each assignment should take to grade, rather than baking that into my formula. (This makes it much easier to reduce my backlog by deciding to “just grade faster” – and to notice when that’s not actually feasible.) Some of the column names are possibly vague to outside eyes but I think it essentially makes sense. Happy to answer questions if anyone has any.

3 Likes